{"_id":"@microsoft/bf-orchestrator-cli","_rev":"281-7148d0b77c740b522517ed83f1e762b2","name":"@microsoft/bf-orchestrator-cli","dist-tags":{"beta":"4.13.0-beta.20210316.e8ec340","latest":"4.15.0","rc":"4.15.0","next":"4.16.0-dev.20231122.4561d55"},"versions":{"0.0.0":{"name":"@microsoft/bf-orchestrator-cli","version":"0.0.0","author":"","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@0.0.0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"}],"dist":{"shasum":"dd132a538e56d2b7b884d85cc1dd0caeea6a19a3","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-0.0.0.tgz","fileCount":1,"integrity":"sha512-PrpV7wLPwKtMOXkWO8+zdl5uX9OZUCOCjR0AHXrw5Qg+tTeResFjOPXegx2CYIVEoh3RHG6K/nMr/VpKGlR3Aw==","signatures":[{"sig":"MEYCIQDGntcSwtHHqJgTl7Qj3XrQiwNgyaZ01YZ+DdiqgxEF+QIhAN+i/9EuRYiwa6A6tl0Qi4BVWZZzARCFhsi6zkKdA+Mw","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":226,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfLeJHCRA9TVsSAnZWagAAcgEP/1oPAaK4kau0+64oZ4GW\nsRkCHlRj4YT2dvpmc8e/h/No7gwtgAffkTlES1T41ettgGFmWDZPzf6yN0wZ\njEI5O3LeAm0uOrEnm3KgHM1XXlDBbvhI7zoh/ds1F19VoVwDx8BYn0NtVj9A\n3Rof0zFYbT8avTDLy3NpabHBjQvWz2rH8PkTFHnXHY/9sQFwMyjSvJzXGmtN\nVbot5OXfvzE4E8zDDEOdL2UmdzoNhi7OY6eLzNlYu/Y3YI/WsnQ9Nk8R8WOX\n8BlCduHhFZoZrAbj8ob1oY5UfMFAPpV3UknMIWJOvJWf/SFoy8xPoVjWtdPy\nYiXDRy2Yf+Q0SQDs2ldGN2X1ERtUd3nqKkL7fsWvU8bhmprB11as7n1imMCh\njud42qpMIXjzf42flGhEkpdYaK60VSv4ObV3PDLYWByb4QBwR5t0QDoYqQ+P\nNEWEtH12xlOv/A1eeW8hYpkTwEaGbIHuHACA3OEvm+K+ZokxLywjJ2Tw8HB4\nnCgJXPNi1N1UUNkJxvlCgUsvQ4HcIM/iTfODxOhiqY1wFdA/fDMHgJqCvLCP\nqLB7uy9q/abkPEG8s7GQgvf+ZamCRAwOJB5vD70WNrDfBcNEM6KifxotKD/u\nJL4JoeanVzwKgW8kK4pnVX2tIMA/GrlsLUzbw0y/kKNma7v+38AOcKa7B5ZC\n1jJb\r\n=KO1X\r\n-----END PGP SIGNATURE-----\r\n"},"main":"index.js","scripts":{"test":"echo \"Error: no test specified\" && exit 1"},"_npmUser":{"name":"microsoft","email":"npmjs@microsoft.com"},"_npmVersion":"6.14.7","description":"","directories":{},"_nodeVersion":"12.16.1","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_0.0.0_1596842566710_0.8284256535216941","host":"s3://npm-registry-packages"}},"4.11.0-beta.875403c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.875403c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.875403c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1d519ae0b4e784e3df0e23d9cf3eae4e1031d2ff","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.875403c.tgz","fileCount":42,"integrity":"sha512-129X3TdvjiDN9EC8URtzP+PhoKSGaKny96wwoOKzPTxFqfWvbrzdsH7wCXvA99QnVbbtFonMMVjq5jRjur15Mw==","signatures":[{"sig":"MEUCIQCrWfkWwJLHVcZUdu9Mfaq4XWgyOjYGgTwPRqvApWF6yAIgRErc4nIjAK7hVtFjDDfZgPuI6mezy37zOAELvGkjDEg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":122050,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNGmGCRA9TVsSAnZWagAAbKYP/iT324z9uqRalfu8v2Qg\nG1SpfpmGVu+NTuTYXJ7foZB27/b6S7376/W4cXzrQbY56MUiidPsOrzQkjtX\n+MW1haA+OEN5ZIW+7/KnRKqGUjlaQZZtYM8IMfMrjWE4T4CVDx1hrYc2xFza\nBbkhVv/1vlHaFF0N2dETeHqJoHaEACAjOeAswI/OX5NjLr85reTOBN0Ay2/B\nVOpE0pafeI5w9QDUVjbFR9ELdHNGZrhOInDoJ733luXbfw/RfY5nP3VyFQhK\ntQd1YadcZh/f3JCXmq3XfBeUIcpoUahlAz2EEwgVz7z84FWItgAfHN95ztqA\n6BFbGNpNe7jUupLXUZG9iyPkwnHDBtcDCH9SJW3i55S2JVeL0zVNtTAPGcxs\n88rtHDO+ugRhwG3Jnh6EvqZe4o4n2QHhlPr9EaCcj7c5pTnuDWJsQeRm/3zU\npNe29+hDSmPm5n5clYdE+LX8TBDW3M+jQ1AbGcRb9EG1H1NXlrL6Vk+660iM\nUW/7BbH+IBC0OTiue4vkIL5kvO/G08P05rJzShWDc7h55Drq0LOg6TyvPEAM\neIFngU0omAaYo2RVW9GEJLiQgEqvCjS6WNxXnFUxo617z2+44Xw43W1YO/pT\nqLJh2AS4CM5mdLvRgDPKZitu+lQsCDuhcChAMkAW24EUUrD5TOtJQYaWM5qv\nnVhH\r\n=kvZC\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.875403c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\n# Commands\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\n* [`bf orchestrator:finetune`](#bf-orchestratorfinetune)\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files to existing orchestrator examples file\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\n directory.\n\n -p, --prefix=prefix Prefix to be added label in snapshot.\n\n -s, --snapshot=snapshot Existing orchestrator snapshot to append to.\n\n --hierarchical Add hierarchical labels based on input file name.\n\nEXAMPLE\n\n $ bf orchestrator:add \n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:assess`\n\nAssess utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:assess\n\nOPTIONS\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -o, --out=out Directory where analysis and output files will be placed.\n -t, --prediction=prediction Path to a prediction file.\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\assess.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\assess.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working\n directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreates Orchestrator example file from .lu/.qna files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\n to current working directory.\n\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator snapshot file will be placed. Default to current working\n directory.\n\n --hierarchical Add hierarchical labels based on lu/qna file name.\n\nEXAMPLE\n\n $ bf orchestrator:create \n $ bf orchestrator:create --in ./path/to/file/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:evaluate`\n\nCreate an Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file .\n\n```\nUSAGE\n $ bf orchestrator:evaluate\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\evaluate.ts)_\n\n## `bf orchestrator:finetune`\n\nManage Orchestrator fine tuning.\n\n```\nUSAGE\n $ bf orchestrator:finetune\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator finetune command help\n\n -i, --in=in If --input is provided, the path to .lu/.qna files from where orchestrator finetune\n example file will be created from. Default to current working directory.\n\n -l, --logformat=logformat (Optional) If --logformat is provided, overrides the log file formats (Supported:\n labelText, dteData).\n\n -m, --model=model Path to Orchestrator model.\n\n -n, --nlrversion=nlrversion (Optional) If --nlrversion is provided, overrides the nlr version (Supported: 4.8.0,\n 4.8.0-multilingual).\n\n -o, --out=out If --get flag is provided, the path where the new orchestrator finetune job will be\n created. Default to current working directory.\n\nEXAMPLE\n\n $ bf orchestrator:finetune status\n $ bf orchestrator:finetune put --in ./path/to/file/ [--nlrversion | --logformat ]\n $ bf orchestrator:finetune get [--out ./path/to/output/]\n```\n\n_See code: [src\\commands\\orchestrator\\finetune.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\finetune.ts)_\n\n## `bf orchestrator:nlr:get`\n\nGets Orchestrator model\n\n```\nUSAGE\n $ bf orchestrator:nlr:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator nlr:get command help\n -o, --out=out Path to Orchestrator model.\n -v, --verbose Enable verbose logging\n --versionId=versionId Model version to download.\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\nlr\\get.ts)_\n\n## `bf orchestrator:nlr:list`\n\nLists all Orchestrator model versions\n\n```\nUSAGE\n $ bf orchestrator:nlr:list\n\nOPTIONS\n -h, --help Orchestrator nlr:list command help\n -r, --raw Raw output\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\nlr\\list.ts)_\n\n## `bf orchestrator:predict`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:predict\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:predict \n $ bf orchestrator:predict --in ./path/to/file/\n $ bf orchestrator:predict --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\predict.ts)_\n\n## `bf orchestrator:test`\n\nTest utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -t, --test=test Path to a test file.\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.875403c/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:finetune COMMAND`](#bf-orchestratorfinetune-command)\r\n* [`bf orchestrator:nlr`](#bf-orchestratornlr)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on a prediction file against a ground-truth file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth .json file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction .json file.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then it\r\n generates two detailed evaluation reports, one for intent prediction, and the other entity.\r\n\r\n The two input JSON files each contains a JSON array following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents\" labels. The utterance can also has an \"entities\" array of entity JSON objects.\r\n Each entity contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for entity mentions, however this attribute is only for debugging purpose and not needed.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\n The \"assess\" command reads the input ground-truth and prediction file and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction labels together, compares them, determines if\r\n a prediction is in the ground-truth or vice versa. At then end the \"assess\" command\r\n will generate HTML reports.\r\n\r\n The reports have following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here is the logic outline:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then the utterance is a TP for this label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity's confusion matrix.\r\n FP and FN logic is similar to those of Intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report. \r\n\r\n Once the serial of per-label binary confusion matrices are built, there would be plenty of\r\n per-label metrics, which can be consolidated by several averaging approaches.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive,\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (positives) from the series of binary confusion matrices.\r\n 1) Summation Micro-Average - the second way condensing the series of binary confusion matrices into one by\r\n summing up theie TP, FP, TN, FN numbers. These 4 summations are then used to calculate precision,\r\n recall, F1, and accuracy.\r\n 2) Macro-Average - another way condensing the series of binary confusion matrices simply takes\r\n arithmetic average of their every metrics individually. The denominator for computing the averages\r\n is the number of labels in the ground-truth set.\r\n 3) Summation Macro-Average - While micro-average takes a simple arithmetic average for every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells. Precision, recall,\r\n F1, and accuracy are then calculated by the average 4 cells.\r\n 4) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some labels completely. Therefore the denominator for this metric\r\n only use the number of prediction labels, i.e., labels with a greater-than-zero support.\r\n 5) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 6) Weighted Macro-Average - this metric approach take an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences calculated in the\r\n first tab.\r\n 7) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 8) Multi-Label Subset Aggregate - Orchestrator support multi-labels for each utterance. I.e., an utterance\r\n can have more than one meanings (intents). For such a multi-intent application, a model can\r\n actually generate spurious intent predictions that can still boost all the per-label metrics\r\n described thus far. Multi-Label Subset Aggreagate\r\n is a per-instance metric that it builds a binary confusion matrix directly.\r\n An utterance is only a TP if a prediction's intent array is an non-empty subset of the ground-truth array. If a prediction's intent array contains one label not the ground-truth array,\r\n then this utterance is a FP. If a prediction's intent array is empty while it's not\r\n for the ground-truth intent array, then the utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions.\r\n\r\nEXAMPLE\r\n```\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator example file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files. Optional.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example set. It also generates a detailed evaluation report with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\evaluate.ts)_\r\n\r\n## `bf orchestrator:finetune COMMAND`\r\n\r\nManage Orchestrator fine tuning.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:finetune COMMAND\r\n\r\nARGUMENTS\r\n COMMAND The \"command\" is the first mandatory argument. This can be \"status\", \"put\" or \"get\".\r\n status - Status of the last finetune training job.\r\n put - Put finetune training example data to improve orchestrator.\r\n get - Get the model for completed finetune job.\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator finetune command help\r\n\r\n -i, --in=in If --input is provided, the path to .lu/.qna files from where orchestrator finetune\r\n example file will be created from. Default to current working directory.\r\n\r\n -l, --logformat=logformat (Optional) If --logformat is provided, overrides the log file formats (Supported:\r\n labelText, dteData).\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -n, --nlrversion=nlrversion (Optional) If --nlrversion is provided, overrides the nlr version (Supported: 4.8.0,\r\n 4.8.0-multilingual).\r\n\r\n -o, --out=out If --get flag is provided, the path where the new orchestrator finetune job will be\r\n created. Default to current working directory.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:finetune status\r\n $ bf orchestrator:finetune put --in ./path/to/file/ [--nlrversion | --logformat ]\r\n $ bf orchestrator:finetune get [--out ./path/to/output/]\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\finetune.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\finetune.ts)_\r\n\r\n## `bf orchestrator:nlr`\r\n\r\ndescribe the command here\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr\r\n\r\nOPTIONS\r\n -f, --force\r\n -h, --help show CLI help\r\n -n, --name=name name to print\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\nlr.ts)_\r\n\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file.\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\test.ts)_\r\n\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest && oclif-dev readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.875403c","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.875403c"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.875403c_1597270405735_0.8672466759469273","host":"s3://npm-registry-packages"}},"4.11.0-beta.6e2953b":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.6e2953b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.6e2953b","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"865bcccc18053d99e1815e6b00d293aeb3d63a99","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.6e2953b.tgz","fileCount":42,"integrity":"sha512-z+OPhWeFhZYO0b/ZIb0Yr1CJ/YNmPmlQgs2p2UelMkjDRWxciXtD6J8JwgPDLEVWcfHq12LA+gW98yqu6Mp2Ig==","signatures":[{"sig":"MEUCIDWEp/wMe62C0ZHp3txwLV6IxW+sJImQnZX4fEGBcOcGAiEAzeaRb3rOwDLDQmC9yCaqyzyaik/6EFouOQrQ4Qgg2NI=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":122050,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNIJgCRA9TVsSAnZWagAAZ9QP/iPOHuSnRvXayA2z5wBt\nFbG16l/kRSUkws2ETYVS9uL6mJ3khE44e4pIHuS/8Xi1Sjnzv9ZXl2a6gjyC\nzCyTd7RRHs4DmS6N1PGpgKV8PmbGF5GzljQ+0ocybCMqIUyjgq3+LuJ/F5yd\nkvRnvJo9DzSH/NR/RWam38+IQy2c53enHBr2XP7b6plZ6sLx4sBnTA+t+iSm\nerPWNPRnzigy/0eUvPWPzufMoMPNxAO6SJnTEklGKlArSsBcEWdk2X9ef3cm\nnsjNEeGPfD1IBO1xXMewDWYFlUeQervO3h3th+aYBaUk7Gdo3Eg+VNLh1M8s\nLE7J3LCCnFiyLXCv+DsBDRjRp3qDyPuzPfaCmRaaGPbYx6owyXko4O7YyJJe\niGW5TGxoZ7kdgSo19eaWM9IrebjRemekuL6HH9O2yTYXeJ9FHLN8qkge7cYp\nx2ZJ7Dl63VQDkdfZDNPxFFAC2gsCVLvOext9ZwxBArWICtzgxpkShZnhJAwI\nGG6Qfr15DOwZ4xGrj/XE3Ug2ORFfp1aAi7a6xNFibxJ61g1Smxn4rjpz7DOf\nilsx3b7ieJDJWYYnYaVkv1fLKKNQAHqtaKmT4Z8Y14GgrDYewYRJ4tmb5oMz\n7sdAyX761N0OHp3hjhcohxi91CtlD1S2zz4TuI0ZubSAiJ//Fn8AvW/jXLUu\nvVAf\r\n=MI1t\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.6e2953b.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\n# Commands\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\n* [`bf orchestrator:finetune`](#bf-orchestratorfinetune)\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files to existing orchestrator examples file\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\n directory.\n\n -p, --prefix=prefix Prefix to be added label in snapshot.\n\n -s, --snapshot=snapshot Existing orchestrator snapshot to append to.\n\n --hierarchical Add hierarchical labels based on input file name.\n\nEXAMPLE\n\n $ bf orchestrator:add \n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:assess`\n\nAssess utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:assess\n\nOPTIONS\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -o, --out=out Directory where analysis and output files will be placed.\n -t, --prediction=prediction Path to a prediction file.\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\assess.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\assess.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working\n directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreates Orchestrator example file from .lu/.qna files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\n to current working directory.\n\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator snapshot file will be placed. Default to current working\n directory.\n\n --hierarchical Add hierarchical labels based on lu/qna file name.\n\nEXAMPLE\n\n $ bf orchestrator:create \n $ bf orchestrator:create --in ./path/to/file/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:evaluate`\n\nCreate an Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file .\n\n```\nUSAGE\n $ bf orchestrator:evaluate\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\evaluate.ts)_\n\n## `bf orchestrator:finetune`\n\nManage Orchestrator fine tuning.\n\n```\nUSAGE\n $ bf orchestrator:finetune\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator finetune command help\n\n -i, --in=in If --input is provided, the path to .lu/.qna files from where orchestrator finetune\n example file will be created from. Default to current working directory.\n\n -l, --logformat=logformat (Optional) If --logformat is provided, overrides the log file formats (Supported:\n labelText, dteData).\n\n -m, --model=model Path to Orchestrator model.\n\n -n, --nlrversion=nlrversion (Optional) If --nlrversion is provided, overrides the nlr version (Supported: 4.8.0,\n 4.8.0-multilingual).\n\n -o, --out=out If --get flag is provided, the path where the new orchestrator finetune job will be\n created. Default to current working directory.\n\nEXAMPLE\n\n $ bf orchestrator:finetune status\n $ bf orchestrator:finetune put --in ./path/to/file/ [--nlrversion | --logformat ]\n $ bf orchestrator:finetune get [--out ./path/to/output/]\n```\n\n_See code: [src\\commands\\orchestrator\\finetune.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\finetune.ts)_\n\n## `bf orchestrator:nlr:get`\n\nGets Orchestrator model\n\n```\nUSAGE\n $ bf orchestrator:nlr:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator nlr:get command help\n -o, --out=out Path to Orchestrator model.\n -v, --verbose Enable verbose logging\n --versionId=versionId Model version to download.\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\nlr\\get.ts)_\n\n## `bf orchestrator:nlr:list`\n\nLists all Orchestrator model versions\n\n```\nUSAGE\n $ bf orchestrator:nlr:list\n\nOPTIONS\n -h, --help Orchestrator nlr:list command help\n -r, --raw Raw output\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\nlr\\list.ts)_\n\n## `bf orchestrator:predict`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:predict\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:predict \n $ bf orchestrator:predict --in ./path/to/file/\n $ bf orchestrator:predict --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\predict.ts)_\n\n## `bf orchestrator:test`\n\nTest utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -t, --test=test Path to a test file.\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6e2953b/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:finetune COMMAND`](#bf-orchestratorfinetune-command)\r\n* [`bf orchestrator:nlr`](#bf-orchestratornlr)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on a prediction file against a ground-truth file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth .json file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction .json file.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then it\r\n generates two detailed evaluation reports, one for intent prediction, and the other entity.\r\n\r\n The two input JSON files each contains a JSON array following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents\" labels. The utterance can also has an \"entities\" array of entity JSON objects.\r\n Each entity contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for entity mentions, however this attribute is only for debugging purpose and not needed.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\n The \"assess\" command reads the input ground-truth and prediction file and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction labels together, compares them, determines if\r\n a prediction is in the ground-truth or vice versa. At then end the \"assess\" command\r\n will generate HTML reports.\r\n\r\n The reports have following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here is the logic outline:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then the utterance is a TP for this label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity's confusion matrix.\r\n FP and FN logic is similar to those of Intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report. \r\n\r\n Once the serial of per-label binary confusion matrices are built, there would be plenty of\r\n per-label metrics, which can be consolidated by several averaging approaches.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive,\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (positives) from the series of binary confusion matrices.\r\n 1) Summation Micro-Average - the second way condensing the series of binary confusion matrices into one by\r\n summing up theie TP, FP, TN, FN numbers. These 4 summations are then used to calculate precision,\r\n recall, F1, and accuracy.\r\n 2) Macro-Average - another way condensing the series of binary confusion matrices simply takes\r\n arithmetic average of their every metrics individually. The denominator for computing the averages\r\n is the number of labels in the ground-truth set.\r\n 3) Summation Macro-Average - While micro-average takes a simple arithmetic average for every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells. Precision, recall,\r\n F1, and accuracy are then calculated by the average 4 cells.\r\n 4) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some labels completely. Therefore the denominator for this metric\r\n only use the number of prediction labels, i.e., labels with a greater-than-zero support.\r\n 5) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 6) Weighted Macro-Average - this metric approach take an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences calculated in the\r\n first tab.\r\n 7) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 8) Multi-Label Subset Aggregate - Orchestrator support multi-labels for each utterance. I.e., an utterance\r\n can have more than one meanings (intents). For such a multi-intent application, a model can\r\n actually generate spurious intent predictions that can still boost all the per-label metrics\r\n described thus far. Multi-Label Subset Aggreagate\r\n is a per-instance metric that it builds a binary confusion matrix directly.\r\n An utterance is only a TP if a prediction's intent array is an non-empty subset of the ground-truth array. If a prediction's intent array contains one label not the ground-truth array,\r\n then this utterance is a FP. If a prediction's intent array is empty while it's not\r\n for the ground-truth intent array, then the utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions.\r\n\r\nEXAMPLE\r\n```\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator example file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files. Optional.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example set. It also generates a detailed evaluation report with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\evaluate.ts)_\r\n\r\n## `bf orchestrator:finetune COMMAND`\r\n\r\nManage Orchestrator fine tuning.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:finetune COMMAND\r\n\r\nARGUMENTS\r\n COMMAND The \"command\" is the first mandatory argument. This can be \"status\", \"put\" or \"get\".\r\n status - Status of the last finetune training job.\r\n put - Put finetune training example data to improve orchestrator.\r\n get - Get the model for completed finetune job.\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator finetune command help\r\n\r\n -i, --in=in If --input is provided, the path to .lu/.qna files from where orchestrator finetune\r\n example file will be created from. Default to current working directory.\r\n\r\n -l, --logformat=logformat (Optional) If --logformat is provided, overrides the log file formats (Supported:\r\n labelText, dteData).\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -n, --nlrversion=nlrversion (Optional) If --nlrversion is provided, overrides the nlr version (Supported: 4.8.0,\r\n 4.8.0-multilingual).\r\n\r\n -o, --out=out If --get flag is provided, the path where the new orchestrator finetune job will be\r\n created. Default to current working directory.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:finetune status\r\n $ bf orchestrator:finetune put --in ./path/to/file/ [--nlrversion | --logformat ]\r\n $ bf orchestrator:finetune get [--out ./path/to/output/]\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\finetune.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\finetune.ts)_\r\n\r\n## `bf orchestrator:nlr`\r\n\r\ndescribe the command here\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr\r\n\r\nOPTIONS\r\n -f, --force\r\n -h, --help show CLI help\r\n -n, --name=name name to print\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\nlr.ts)_\r\n\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file.\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v1.0.0/src\\commands\\orchestrator\\test.ts)_\r\n\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest && oclif-dev readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.6e2953b","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.6e2953b"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.6e2953b_1597276767773_0.10723439388675726","host":"s3://npm-registry-packages"}},"4.11.0-beta.6afce3c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.6afce3c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.6afce3c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"01ffe2ac8c778de1f6ee8ca2e6c1ab5de04baaf9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.6afce3c.tgz","fileCount":42,"integrity":"sha512-IR2/BbEAo8ohY7I7WnPFBMqbOk/eNjstxOl3rvIBbNO1VNtrr7VNih0/5r4ndrDUMHPVKb+yaVOSHOZgFv5NMg==","signatures":[{"sig":"MEQCIAeb8fZb3gV8ilwQR3nsuE1wMzuN8HPn6p1A3EHtjFYlAiAFNZC7AqGs6tiyhP/A37ESeJAoy0it9LRhEsZk96ErTA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124759,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfNYtsCRA9TVsSAnZWagAA5T0P/R9RvdvY9Sfz2ZC3Xn/p\n+mCnhyN+bOEr0M5QfQ0B/jFAUZPyaMRz7ORwmA4T8hMUnKxtIoSxxmLYwRFJ\nLTMGF00Q0dV1++eiUvbkGPwxg/KAKz6XSWChWgMbewKosMGoaphvhOCkojLp\nW+fGQ2ETAha/RC4R6hDSVfib0pT4g3+1KgfguWw9dJ6nD8ApQFDXtZF24aen\nBkoTQBY9PTQzFwxhGPNRAfCwwVacFxLZo8ouPZy6trb/fHkX94XUdsOvcGkJ\nyewF1uQUI+X5gPKhpCVD0E5iNMIZa0E8k0wL2Yqd9sUu9Ub6rMnVDmKGOnZS\n5UdGEKA7OiYh3F5rhaQY1M0s7V+k52EJK+6gz8xVDPxDF+JiJ4MO0v/2L+0x\nce54ufhrk0WsEG9Vh1C+K3gRWJuHXDsE3FnOAY7U7O/qQERUR2VM8osGeMDH\n2vtrza8/ShHtHgLGtR0swNBIi4lYx2+f3dj2iI5AJfJyLjGs2uDHNBIOpiqe\nVpWITrRyHz1mIZTAGYATB4pn3WtGe0QqhAfTOFJqbcNAGyA1f5kPQ3xTYxWX\nPg0uU81PQ1X3dkw0lS9X/bboxL6KLIwRtkhzaJHL0SrgNGoWjA/VgW8MNjFx\nWVqZtolCvSrvyqibH3TXKWwskUadvElIRrx8ldVr9ITWrQ6ZFXGCL0PzKpCm\nWTCj\r\n=ZnQK\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.6afce3c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\n# Prerequisite\r\n- [Node.js](https://nodejs.org/) version 10 or higher\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n# Installation\r\nTo install:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\n* [`bf orchestrator:finetune`](#bf-orchestratorfinetune)\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files to existing orchestrator examples file\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\n directory.\n\n -p, --prefix=prefix Prefix to be added label in snapshot.\n\n -s, --snapshot=snapshot Existing orchestrator snapshot to append to.\n\n --hierarchical Add hierarchical labels based on input file name.\n\nEXAMPLE\n\n $ bf orchestrator:add \n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:assess`\n\nAssess utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:assess\n\nOPTIONS\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -o, --out=out Directory where analysis and output files will be placed.\n -t, --prediction=prediction Path to a prediction file.\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\assess.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\assess.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working\n directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreates Orchestrator example file from .lu/.qna files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\n to current working directory.\n\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where generated orchestrator snapshot file will be placed. Default to current working\n directory.\n\n --hierarchical Add hierarchical labels based on lu/qna file name.\n\nEXAMPLE\n\n $ bf orchestrator:create \n $ bf orchestrator:create --in ./path/to/file/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:evaluate`\n\nCreate an Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file .\n\n```\nUSAGE\n $ bf orchestrator:evaluate\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\evaluate.ts)_\n\n## `bf orchestrator:finetune`\n\nManage Orchestrator fine tuning.\n\n```\nUSAGE\n $ bf orchestrator:finetune\n\nOPTIONS\n -d, --debug\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator finetune command help\n\n -i, --in=in If --input is provided, the path to .lu/.qna files from where orchestrator finetune\n example file will be created from. Default to current working directory.\n\n -l, --logformat=logformat (Optional) If --logformat is provided, overrides the log file formats (Supported:\n labelText, dteData).\n\n -m, --model=model Path to Orchestrator model.\n\n -n, --nlrversion=nlrversion (Optional) If --nlrversion is provided, overrides the nlr version (Supported: 4.8.0,\n 4.8.0-multilingual).\n\n -o, --out=out If --get flag is provided, the path where the new orchestrator finetune job will be\n created. Default to current working directory.\n\nEXAMPLE\n\n $ bf orchestrator:finetune status\n $ bf orchestrator:finetune put --in ./path/to/file/ [--nlrversion | --logformat ]\n $ bf orchestrator:finetune get [--out ./path/to/output/]\n```\n\n_See code: [src\\commands\\orchestrator\\finetune.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\finetune.ts)_\n\n## `bf orchestrator:nlr:get`\n\nGets Orchestrator model\n\n```\nUSAGE\n $ bf orchestrator:nlr:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator nlr:get command help\n -o, --out=out Path to Orchestrator model.\n -v, --verbose Enable verbose logging\n --versionId=versionId Model version to download.\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\nlr\\get.ts)_\n\n## `bf orchestrator:nlr:list`\n\nLists all Orchestrator model versions\n\n```\nUSAGE\n $ bf orchestrator:nlr:list\n\nOPTIONS\n -h, --help Orchestrator nlr:list command help\n -r, --raw Raw output\n```\n\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\nlr\\list.ts)_\n\n## `bf orchestrator:predict`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:predict\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:predict \n $ bf orchestrator:predict --in ./path/to/file/\n $ bf orchestrator:predict --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\predict.ts)_\n\n## `bf orchestrator:test`\n\nTest utterance/label samples from an input file and create an evaluation report\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -a, --ambiguous=ambiguous Ambiguous threshold, default to 0.2\n -d, --debug\n -h, --help show CLI help\n -i, --in=in Path to a previously created Orchestrator .blu file.\n -l, --low_confidence=low_confidence Low confidence threshold, default to 0.5\n -m, --model=model Directory or hosting Orchestrator config and model files.\n -o, --out=out Directory where analysis and output files will be placed.\n -p, --multi_label=multi_label Plural/multi-label prediction threshold, default to 1\n -t, --test=test Path to a test file.\n -u, --unknown=unknown Unknow label threshold, default to 0.3\n\nEXAMPLE\n\n $ bf orchestrator:evaluate \n $ bf orchestrator:evaluate --in ./path/to/file/\n $ bf orchestrator:evaluate --in ./path/to/file/ --out ./path/to/output/\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.11.0-beta.6afce3c/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:nlr`](#bf-orchestratornlr)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on a prediction file against a ground-truth file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth .json file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction .json file.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then it\r\n generates two detailed evaluation reports, one on intent prediction, and the other entity.\r\n\r\n The two input JSON files each contains a JSON array of labeled utterances\r\n following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n will generate HTML reports at the end. One report for intent evaluation and the other entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity mention candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance. It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays.\r\n However as some important metrics such as precision, recall, and their combination, F1, do not rely on TN,\r\n so it's really necessary to calculate TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be consolidated using several averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive,\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 2) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 3) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 4) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 5) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 6) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 7) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 8) Multi-Label Exact Aggregate - Orchestrator supports multi-label intents for each utterance. I.e., an utterance\r\n can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds a binary confusion matrix directly and only a instance can contribute to metric computation disregard the number\r\n of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 9) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\t\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\assess.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/assess.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files .\r\n -m, --model=model Path to Orchestrator model.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator example file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files. Optional.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example set. It also generates a detailed evaluation report with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/evaluate.ts)_\r\n\r\n## `bf orchestrator:nlr:get`\r\n\r\nGets Orchestrator model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:get\r\n\r\nOPTIONS\r\n -o, --out Path to Orchestrator model\r\n --versionId Model version to download\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/get.ts)_\r\n\r\n\r\n## `bf orchestrator:nlr:list`\r\n\r\nLists all Orchestrator model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:list\r\n\r\nOPTIONS\r\n -r, --raw Raw output\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/list.ts)_\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Ambiguous analysis threshild. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file.\r\n -u, --unknown=threshold Unknow label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest && oclif-dev readme","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.6afce3c","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.6afce3c"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.6afce3c_1597344620450_0.22969869314539415","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200818.8213b2a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200818.8213b2a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200818.8213b2a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"2f6b1f9f4cf1fc3e1c788c13e693e608898ae6de","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200818.8213b2a.tgz","fileCount":42,"integrity":"sha512-7vp4o7WdqVfsJgSvJ3Z7QuGrIeHlF9IsjNEl7O5e27JUQa6OZSsE+HdW56YzJxLWTVtzQKAQhwipqwBGAr7g1g==","signatures":[{"sig":"MEUCIDvsyaJ4QBmoXuB6NmUn7Wyn5CdIsRBNn/MfRSvbyDVjAiEAmQYwD0u5Q9usJ7VCc9dxDxohh1Gpm1WR/HV0S74sjqg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119678,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfPDgZCRA9TVsSAnZWagAA+cUP/2d5c7zBj6sm4TCAeuyH\nOiZh+T36Q1RkYQd94IEyhYluhqAB8exlPr0njSUvdNj2GgUCLEeMVjiLxCsy\nMLq2AL8xNksOG52oAUjtX8ZhDJfOlKeupe8OvhJTBzI5QKNpNWj71DZRrK7Z\nf05ECnVCFKUwwHGX75/P9ORtHGbzF4I4NERvFTOBLDDEBbEPJtqhjNITB8YJ\nEWTU7wwIrd3aYfJnY7NtSHdyhQwtmj/MueZqvmHgpGcTWwk57IKxAGxL7nXz\nvNznJrTroCpXSi+4fQygKg1tWknGqbL1+mkaYvdSA3GEsxZXG6DHQDLtzSs+\nkeudxc06aIHqjsorW4L6pgkYWL2vNAZLIx16jo/NgYaZenPlreWmVN6ADdda\nXfMqBVmmodPW7hSUvRvj+8LWJQ5qsrOqyYzhp679/jjwXoQXupAFAdceLtRQ\nhiC4fzecrCA+FyqHY8rwfyK7juGFN4U95vVshPI22oyBG4dafsHsRO4Va+Pt\nxDG4S0jFjI9BH+QdQoTRgk1oI4rpK/jrcq9JgOD4ojR6z2lotOXDWx67woI2\nqmt95xV6hSBjDL2cz/74y2OY8gyagDKNOAyl6bXsqSrds5qiGMo8uILN7ZqN\nd2YKDl1wYVjTMbxr0uRqVjzL2DPe3jKPXjvgRRbHUHJsnQmid8OL8sIg9TyB\nyO/b\r\n=4615\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200818.8213b2a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator\r\n======================\r\n\r\nThis package should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\n# Prerequisite\r\n- [Node.js](https://nodejs.org/) version 10 or higher\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n# Installation\r\nTo install:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\n# Commands\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\r\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on a prediction file against a ground-truth file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth label file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one on intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a JSON label array format described below.\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assess\" command aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates as many average/aggregate metrics as we can think of.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholder.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TSV or a JSON array format.\r\n The TSV file format only supports intent labels and it must have 2 columns, labels and utterance,\r\n sepatated by a TAB. The label column can contains multiple labels delimited by camma.\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity mention candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance. It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays.\r\n However as some important metrics such as precision, recall, and their combination, F1, do not rely on TN,\r\n so it's really necessary to calculate TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be consolidated using several averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\assess.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/assess.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files .\r\n -m, --model=model Path to Orchestrator model.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator example file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/evaluate.ts)_\r\n\r\n## `bf orchestrator:nlr:get`\r\n\r\nGets Orchestrator model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:get\r\n\r\nOPTIONS\r\n -o, --out Path to Orchestrator model\r\n --versionId Model version to download\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/get.ts)_\r\n\r\n\r\n## `bf orchestrator:nlr:list`\r\n\r\nLists all Orchestrator model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:list\r\n\r\nOPTIONS\r\n -r, --raw Raw output\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/list.ts)_\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file.\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200818.8213b2a","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200818.8213b2a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200818.8213b2a_1597782041569_0.3605175688567359","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200821.21f8e76":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200821.21f8e76","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200821.21f8e76","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"086eb7862071c42dfa6fffeada7bfe39a7f99715","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200821.21f8e76.tgz","fileCount":42,"integrity":"sha512-lVmleUFWYQdXnncF8WXWxuCVYy2KfIqRyCWQZNbWKS7CxPDkrPrFT68lTxsburc6X0ZxUTZwL69JAJAZ1Qpu+Q==","signatures":[{"sig":"MEQCIC5jtvkcwwOs8BdecXZePGNFMe9AtgLZPB5/xEnQC4TCAiBBhjsuejATNCWLoQz+26x3cQ7wzCq2sNy98cI7X7e+9w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":120159,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfQBbOCRA9TVsSAnZWagAA5wgP/0iIo1at/qCsKb07DawA\n0LxrNnsYmUN0+Sw1+/offSPKIXmphpD8Xr4nCeE6r4Y/t6N/COJIZkRBiZ/O\ngxc8+Ss0zixbjUpCoHa7Z9pcZC6eaapNHfi58vmuQ9tyk5aif1tBOvxTbrUB\nh834Om8TLXMCnFrB3by/he6a8P9V8luvUhY9N79Wk8el+Vi4iVCLOpqb0TBW\noq41dJfe/G+NrMaoKY12BCAD1C3kIhIIYGOtlRPKZXIvlRU2vT2xUdt/OgSn\n/a9PoU/mSgn/JBy/EtfE1YFax36ALLRoNVb3w0qhIE9zmkSJvdpWh1K/jOlh\nwB7QH1oXaALM0goI0JzrJrCgTVa5ndoDIdLe5Izs7iMERPYnEp22DqxvojCV\nZldOYltW4jD1Yi+I9N15L3w3rlMvMwTl3/NrVpO7gia988SLYwzD2ig64r1i\n89VEa80oUckLsf2YFbIuF959fGXBvgTlrOGerHfSYb8YVDAQjjFW42heJ8wq\nVFjf4F9ILqh5XhR3SC+2TP/UQOPWjOA+oE7SR3nByA08+fKk95ryF7/NEyol\ntN25uOLnq0CS3S6OIOTGCMUuM2a5NXgPH+2kPbsTEPOJ+v4TphSL0+3hQPGr\nViVPKT4hufavCT0OQUz3h56Lct0NpnJN2cy48LcueK6eGVxgayyOVLjhR6q3\nIBmH\r\n=TUSZ\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200821.21f8e76.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n\r\n- [Node.js](https://nodejs.org/) version 10 or higher\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\r\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on a prediction file against a ground-truth file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth label file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one on intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a JSON label array format described below.\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assess\" command aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates as many average/aggregate metrics as we can think of.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholder.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TSV or a JSON array format.\r\n The TSV file format only supports intent labels and it must have 2 columns, labels and utterance,\r\n sepatated by a TAB. The label column can contains multiple labels delimited by camma.\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity mention candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance. It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays.\r\n However as some important metrics such as precision, recall, and their combination, F1, do not rely on TN,\r\n so it's really necessary to calculate TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be consolidated using several averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\assess.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/assess.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files .\r\n -m, --model=model Path to Orchestrator model.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator example file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/evaluate.ts)_\r\n\r\n## `bf orchestrator:nlr:get`\r\n\r\nGets Orchestrator model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:get\r\n\r\nOPTIONS\r\n -o, --out Path to Orchestrator model\r\n --versionId Model version to download\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/get.ts)_\r\n\r\n\r\n## `bf orchestrator:nlr:list`\r\n\r\nLists all Orchestrator model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:list\r\n\r\nOPTIONS\r\n -r, --raw Raw output\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/list.ts)_\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file.\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200821.21f8e76","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200821.21f8e76"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200821.21f8e76_1598035662167_0.23888670742332407","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200824.6c63bf3":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200824.6c63bf3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200824.6c63bf3","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"a5ace4e26c648253098d62d3f3dd019a26d351b3","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200824.6c63bf3.tgz","fileCount":42,"integrity":"sha512-mfQgQyGgWcjYPnFlmfirqJbtXv2s+OVRFR/ckAHEo1tJwKaiDqCi5juDwjCq5P64tIUFRuC8ENCyZWxR0mHL0A==","signatures":[{"sig":"MEUCIDY0HnhQFjqh1ECxms5HCETWkpiVJs2Ch3QMqEBlze09AiEA3E1+yjyJ1dGGFlYrpTK59eBRQBBFzAfwJwHBj4yb4bs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfQ2eECRA9TVsSAnZWagAAwfYP/R5wGF6ckqg96e5LGAfJ\nruefq5QSgwwiKgYDVEm/2b/ZeiLuxL1smIVoK+ItUcCmGOeaaDIlb7qmojLq\nuPePGLh2ZxQl7is4I/wRGJkc5ikEhXcpDWsBL7mkzVMZRvMnC31JMLJn48et\nT2X0Gom12IJ09ISaxjQtU/cJsHL3oEFsiq0YhA4yvYu54Tf6T+l9R5gSgxel\ntp2UFQ1EnICb4U4xWoFKoEyNbwWLhkc+CGGFadLVIVtN7pjbLQc9HMdAlp3J\nr+xLhzvq1XZcnZbWdvENsL9q7ybM34hrMLTLe4FrIACifxMRZqTwNJ/XDfm3\neN/yp3c+7cNM9UkK6Y88QzPYweTOZA0xb21Fm4X+SXokHcE1k1Oh81yCVUko\n8oRLgUbVawXV5m/NmH239ACyZq8tZHOMxUF58i4mtF2pwwouip2V4UoNIx2P\nxugw9UiHgTG/9xQW/OQEL1IbvE01OOQO8jUsKtRTY5xU0xtvi9WRjRAF/VWi\naNUsKPVo/FK4e6tOsXClustibbwGyPUSRXTOQw0lGvXaEgk1wO2NF9wUXASj\nuEDHTr2W5r+dV7TmXrQ09cInEyclWRdeHBdKIughSkiy2PY8I5nldmUEMukI\nOeEYGjJ/ediQgTTUmNE6yJK0YNU+6g11uY2hmV65qqLIFTgm2ldGzby4REhB\nwZie\r\n=ND5N\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200824.6c63bf3.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n\r\n- [Node.js](https://nodejs.org/) version 10 or higher\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:assess`](#bf-orchestratorassess)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:evaluate`](#bf-orchestratorevaluate)\r\n* [`bf orchestrator:nlr:get`](#bf-orchestratornlrget)\r\n* [`bf orchestrator:nlr:list`](#bf-orchestratornlrlist)\r\n* [`bf orchestrator:predict`](#bf-orchestratorpredict)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:assess`\r\n\r\nCreate an evaluation report on assessing predictions against ground-truth instances.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:assess\r\n\r\nOPTIONS\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'assess' command help.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assess\" command compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one on intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a JSON label array format described below.\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assess\" command aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates as many average/aggregate metrics as we can think of.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholder.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TSV or a JSON array format.\r\n The TSV file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there are too many possible entity mention candidates for an utterance, while there is only one or a small\r\n number of intent labels for an utterance. It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays.\r\n However as some important metrics such as precision, recall, and their combination, F1, do not rely on TN,\r\n so it's really necessary to calculate TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file may contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be consolidated using several averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\assess.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/assess.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot (.blu) file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:evaluate`\r\n\r\nCreate Orchestrator leave-one-out cross validation (LOOCV) evaluation report on a previously generated .blu file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:evaluate\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'evaluate' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluate' command can execute a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - intent and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple intents and exact utterance/intent duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other intent predictions whose\r\n scores are close to the correctly predicted intents. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted intent score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's intent labels were not scored the highest.\r\n > Low Confidence - utterance intent labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"low_confidence\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\evaluate.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/evaluate.ts)_\r\n\r\n## `bf orchestrator:nlr:get`\r\n\r\nGets Orchestrator model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:get\r\n\r\nOPTIONS\r\n -o, --out Path to Orchestrator model\r\n --versionId Model version to download\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/get.ts)_\r\n\r\n\r\n## `bf orchestrator:nlr:list`\r\n\r\nLists all Orchestrator model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:nlr:list\r\n\r\nOPTIONS\r\n -r, --raw Raw output\r\n -h, --help show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\nlr\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/nlr/list.ts)_\r\n\r\n## `bf orchestrator:predict`\r\n\r\nReturns score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:predict --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'predict' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'predict' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n\r\n The 'predict' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'predict' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:predict --out=resources\\data\\Columnar\\PredictOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"LoadModel\": true,\r\n \"VocabFile\": \"vocab.json\",\r\n \"MergeFile\": \"merges.txt\",\r\n \"ModelFile\": \"traced_model.onnx\",\r\n \"Version\": \"1.0.0-pretrained.20200729.microsoft.dte.en.onnx\"\r\n }\r\n\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\predict.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/predict.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -a, --ambiguous=threshold Optional ambiguous analysis threshold. Default to 0.2.\r\n -d, --debug Print debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -l, --low_confidence=threshold Optional low confidence analysis threshold. Default to 0.5.\r\n -m, --model=model Directory or a config file hosting Orchestrator model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --multi_label=threshold Optional plural/multi-label prediction threshold, default to 1,\r\n i.e., only max-score intents are predicted\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n -u, --unknown=threshold Optional unknown label threshold, default to 0.3.\r\n\r\nDESCRIPTION\r\n The 'test' command can test an Orchestrator model and example set on an input utterance/intent file.\r\n\r\nEXAMPLE\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200824.6c63bf3","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200824.6c63bf3"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200824.6c63bf3_1598252932298_0.6129247592629159","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200904.e677869":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200904.e677869","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200904.e677869","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d6f2d231b57768b8af4cc4b35ceed7c6098adaca","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200904.e677869.tgz","fileCount":42,"integrity":"sha512-+u3nbn1hoZxDzriQSB0i8rinAk8KpJCJY9tnjHVUVZXz+QoF93JbpgJ5yELbFn8g5z2P1wkh+/1BBsrIr0ibHA==","signatures":[{"sig":"MEQCIChTbVNpReRqBwJojmj70Y1LvfAw0f509T9IdJoFi/ynAiBmNkAvLcPT6bNaJoPpy2pdp8FiKhGH3Q3DVSWt7qJG0Q==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfUegcCRA9TVsSAnZWagAAzNwP/iSUU4cl6hg/U5J1rTRh\nGxOCRe7NQ54DfgmM2NWUbIQCSC4ZXjuUUiKAqS405EtvUo7hetathYFdCRRg\nv1FljEKak8iEPCYrL5G5AlGD5Eq/ZlriNt5C0TQJ6yrF2sLqWSWwH1+9shhR\npjtgYf0cKdR2thn5oXk1Ab8BN/y6K3SoDC8MkJd0/+8n7sXvKhxW5sJDYuTy\njn6nLlGUY4OEtxeY0rK1FPwcR9BgFoY+09OAaUraVV6ovM30rLy4s43BijpA\nbVITVzaguA3H3iOMwW2RtWweBo/KU406vNnHF5AfuNI4q1wJGuIj6mnERN5t\njn0cdS+zRXgHugR9FKmaJhhGLRvd/IjL7TIgYaUM7VEXgsrmSboOSglVh2Mn\nj3I9KflHYVVUMEhL2GXv8hEAXJsGmz8IDHJJ5ZY4BPClskJAQkF24U/8OLor\nw4Rde1dIQawJlnK7P1PBthLc9mKHFvl0Sz5Qiz036Ybayx6ycsBKZB1pkAtj\nYrcknZPP49iexdVl6BrQsJg5foeZg5ugjt9IooATlPtqGLhWYYQAIcQ/UCri\nqwv1fRnyDWR/7vIezStashNVctGwQA8W2zysjU3zFJ8YOyVFs7KlrdIFVrhe\nW9DPtzTJ1WbUEhqUrVhsLpaTatA6wPzZ4KfFtIst0U2TrrRWoInTYww2d6L6\nRr+Y\r\n=EIVy\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200904.e677869.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200904.e677869","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200904.e677869"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200904.e677869_1599203355496_0.5107634382219024","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200918.ad209cb":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200918.ad209cb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200918.ad209cb","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"37c33614b1fd5b0dd160bfedafa4e96953a4856f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200918.ad209cb.tgz","fileCount":42,"integrity":"sha512-oI1ZZw4E5POjE37+2hgejLndtxr60Mbux3YGh1zPc2CRzY88pSWa2wGCVed+OJ02mHZtzRrRXock5UkMybKqgQ==","signatures":[{"sig":"MEUCIG9zydKCIIJkkUTPZnWb5XHoXdrOPvrDxf0FiRRR4W8oAiEAvWAqj7N8lmGZXyC0OBMOk1HgvG3VRn7H8AQemmxrhf4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfZF0uCRA9TVsSAnZWagAAlzMP/jMxT8NVSWoQ4BfAUCOa\nd6BOqWshYodUvXAPCaK7BrFNIDtq51niid1Uj1zfMq0Ri6qLAKbbINAMVhAM\n2GifHwfF9fVDM4Ec3wX8RnY6lNLQb/bOcAltKQ+HWNNNM1PhewgvCg7B018S\nJC+R7eJnO2q94m/AZM4PoMieHJfI/bqIWvs34azWb0UQKdMiWkV81brSHPDR\n6Zm7rGB6CTF49tG79slAa45JvSPU1kw4uTEFiABoMxhTZZEXrxpVd0ee2eue\nbeEtDMXSnQtVQlEjTO0foI1ekkT5QODpZ5nwuhF4RlPJmxN67Gu7uw/MVomr\n00SlzXwaxBlD9W0ignl9dCDH7bqgPbggQa9jugM0PE9V7Oalf/wELMkCerAV\nY32f2P23RMeqLBOppniIYs7dhX8EM938So/I6x4+ZGwzFCjE6ZEndLcfMKNN\nOieT5NbiF+WEiSwHPCBim59ks8ZXFgdK9yhRuT7W61e4gvyH3ldhlaSX9M+H\nf0FyV3246t8OLFtf/3brVG3OxmejztB0B3+OfU9gsnMz6yD2+9vaVPjzh/oE\nDvTFcIx7YJEtewQQepalWalYDNrhy+xPRNmTItRRz4A4booe2mtGJDy9XcbG\nsPtpJqOqX0566ej+HrIkZNL+vIClEE5YPZ7aisq10705JXfhnkxUI5YtdNIK\nlyj5\r\n=AsFo\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200918.ad209cb.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200918.ad209cb","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200918.ad209cb"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200918.ad209cb_1600412974371_0.4641433655203344","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200919.b7fef29":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200919.b7fef29","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200919.b7fef29","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"aa923bfd19f91dc2d354150ef5136b79f6694155","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200919.b7fef29.tgz","fileCount":42,"integrity":"sha512-U3xf3ffjX1AuFe1LW2C2UyWRasMgFZWOg+VevLiiCBHs+AAK0O8ny10fFBoUpE6iC1xM/v200P57n5ot7vLBmQ==","signatures":[{"sig":"MEUCIHjPCyvgc7QWwv+fjwIsYhhlbs1P3so9nBl5Jy22rVe7AiEA/dEfSb2sZ1m0tQCSSUGsGmz+Adb5UXbejEwyqyhekc0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfZa5pCRA9TVsSAnZWagAALOsP/i7+stOatOzdWkYLq/fi\n12eeB8fUi+7vygEPXSEE5VtLkN2aG32oiyYZL21IULSVMo8fsWayOZcn6EOz\ngqYD4A8LajSjxurrt4A3CDj35vwYVPeXQHDkYx8qBbUH+L509JZCa6kB0nRt\nba9/1+hPgJze+J9IqbF3erMff1JfnpGyhwG1pnAhFUcvMI2juh+BAHfnSypT\nARq7Y6NSK5vZTFGEAmIjur+t7boMpVd5NgF8x2eaRlfQNnHqRWYXpteaSq6K\n7OBid/qN0sQPFVdOropx1rrQ5IDP0/v1vQtl9AuxXC+PbmWKRHMTn/nkxWU/\nVM9NsqLMVqhXr4aX9NVBxY+Aoa3khJHfkWDPxa5YUUl9cevxeG0HHMfZYCFD\nrigBS7O1yz1xF0eVu3oc2EQU/RwZBo92L5mv02OGi94/9elFP1m72vQHKSSe\nx1AnRuS6p/QOeWmLe8s6g1SBK20QjFVuV2aZIygzgE5V9t+P4SKJq4mR0iP9\nLScfs5B2Hsn6BLxvuk0QKD1Qo5GJM0lJgtrE5Hnxx/MAbql7i4DEVNZsrfcW\nuFlVE2QiY/JXp+om3zWLT6/PUHkNpECyNt3kofDKMn9I+azrvs5+yQ/xDc8x\npWKjxGb9qPFokd/HSsFx7VZJAu1wkZ7ulGJLoVWgH2LjVPNDMwh/NEnEsKYt\n7paU\r\n=RFr+\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200919.b7fef29.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200919.b7fef29","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200919.b7fef29"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200919.b7fef29_1600499305224_0.4503836964203345","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200921.2320a67":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200921.2320a67","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200921.2320a67","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ef398597b813d92e6f6fefe99bb3055e9ffbabf4","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200921.2320a67.tgz","fileCount":42,"integrity":"sha512-UhzkY6GhB76BCQKpf1lQjSg+NS9TPQ0KTMQSarhtPqRr20bCf4Y38xLehCofn0mvfCIPXrHo7ETTtugm8KXSBg==","signatures":[{"sig":"MEUCIQDzMivaR66DmMxpFxYau1BHjjQmI5FIKpkTEFLRqZXW8gIgHsE6DPbC5dDQ3dJ2V+ALLGmFxbft7eRrkpdZDj5Qszw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfaFF+CRA9TVsSAnZWagAAatIP/1QiYf1gjM6iKNz+o2Ht\nTc/8inRvZ/oLNpWdDSG22UCHk+M2plEsEEGtfYoPOPfkLOPUX6mhLXkPJbGm\nrT6qZzFElq4EI7jdRWpUFPDeCdBYdG3jSDvKu59QXXg0Zu0xPTj1oE5XlEec\nRZlPTzU9KmMwad49e8kP9ltnNrkxvgbb+faBQB/qavG1SKtUrESUysgXWxsb\nfsQ9oL0jJtyl2/mt932dFYxyz+Bz13SNg6gU/UNi8smDLhChGhTbwxhRYqW9\nbb0DnzhhUuAP+jhKxZrIP+HI9onLzC9DXayHkck5r7KuSmPdUeCeOJbqPbRp\nxpGhNdLhnMbGtUngGoGGYq0OtDyK0dvBbA6W68J7r8D84fHzOAFjXAcxvVz5\ngfh3aI5lStFOZlGW1yLnnx4CL4AWqr1QgRJECbmM5wcW2Zmrt9nJFOoCCG8K\n4kxU46LqMoTW1jMhdJsLtcLWKdsvqZDOQMZf3TrcwV44RYcBWqtXYJCIeSV+\n3HLQLUiRj4RPEXxwP/hGez6o90etZ5boqadGIJwrQuYH6BpdVAOFdTUB4ZtL\nK2R6DHUh+23BLsKOhoLqKfsbcydvIVfPaNQozaO/QElYEHuFgILYoSqZXXw6\nKTpqb/YWJP+YjDODQCmFdiUBtEzLC1tihKMrq22Y9QAn8yZ2l8bX/vsU5zO8\ngci1\r\n=djvY\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200921.2320a67.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200921.2320a67","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200921.2320a67"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200921.2320a67_1600672126221_0.840242348383824","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200922.a071f5c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200922.a071f5c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200922.a071f5c","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"9c9a80e38a13deffc7ce00bf7efc660c19cc2cab","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200922.a071f5c.tgz","fileCount":42,"integrity":"sha512-rTJJRESrLdB19AtievJu6GC9RSUHBXrdDErA/2fZL+7A6HAQh3DPJYlSVR/AjF2faf2IlumXW8j+IvvJYjOoqQ==","signatures":[{"sig":"MEUCIQC9U8LkwMH7MU8zPQnt8sZ5aI5L/nLq6L5/UsbusDn20gIgbEzhr65Hj0Z0Q0tV3G4n8rqGkrAVaC+eXUFagmcmnxQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121352,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfaj4nCRA9TVsSAnZWagAAaFQP/1nDWTzCC12L7YXh6EKN\nkGF5VhozSMsMNuhyQ7Ze6Coa6Q0IOOytb8vPl/N8BrcmbDY66u0oKZzlBx+f\nG/MEQ071zxfgog2ot20SNCqAuae3ZqTUNs4MoXpeEZ3XnY4QWqTbc6M47208\n2nbf56KI97s5NDO8OQR+BFWszsEPEa7kaKABFx4PZGUfRNXz0QX+5i72v3qZ\nuOVg0NWqmsnsLZMTQbi9uW861w6oJkRgCuIWwyr7wmuCXWVGUiEcR3BRlz6T\n7wsCMROgPCg9qUhPzt7cNAfKUzm2v4AN78AJhllGUT+nE7t1RVHY+vlhiC2Z\nt0IvyECfAUaWef8k+TdSDGD5viSvWC+ZGd7QBwrjlmZckUeUQ08kw8Dso9wq\nvfmZ1rawfcIY/Pg0Xsusx4TnjfdzZ/OeYlb7OqGQokQw/p0ZRMVupcMjr6WG\nYCXjQxO8fCJHltaV4Ner5896NwjSnIMcvebxPzaTjXnxPJwvCB2ThV+6cw8o\nsawdcaXd0GHIavbiEb+RxDFpyyYfF6Kl7gYHbpXM4gDyuTm7jMhBRN9x0O2G\nt0n97dC5La/WtC9463QO8b0uE2RJZQjDqoLwFTb3YLWDD9adiTvt/EUAHApL\nE88j3zAAWgp99MHmOO6I84W5nRn7jiR0eY4pkgT6MwQftw+1N/qoY/pxf/ka\nLVxR\r\n=S8av\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200922.a071f5c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@microsoft/bf-dispatcher":"4.11.0-beta.20200922.a071f5c","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200922.a071f5c"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200922.a071f5c_1600798246799_0.4515628490651544","host":"s3://npm-registry-packages"}},"4.11.0-beta.20200923.7e94eb2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20200923.7e94eb2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20200923.7e94eb2","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1a2928a82bdf03d7f410694d25e4ddad64b3b14c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20200923.7e94eb2.tgz","fileCount":42,"integrity":"sha512-Ua7PzLhK6nBtsFtoskkMI91mWIO3NgbbecAJZoPeFGXSn9fJbXcQmoa/qnCn6mtwFDziZ2kqvMLRbbjHKDak7g==","signatures":[{"sig":"MEUCIG/uAM3rmWfT3+xASfqjlxfv+HvHl7ZmMmsgrG9/08qqAiEA+gzk+8RcPZgkJ0yhF6GhcY1obRT5/c5R947CKE76j5I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121382,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfavSpCRA9TVsSAnZWagAABFAP/1KQTyyhApQjOqQp/4zJ\nCpp7U6clRzsOHFnEvS4kk7MbaJKaVyV+tBvjsYJBAMbPp9oOVCWUU3yv2FDb\ntQAvplIVF2QSHlc65vzKaULQGIyElIJO0Y9zAvzw0bVGk2yftLCrYlVeOA97\nK877sgz/Vy1457MnGnPLBCYexsFvQwSSeHBFNDzmwvGOleZU4hJfUJ+1xO7J\n4rRJC0A50fXlnQM3yGc4kGN/G/p8QV7/cXDO7rD4w5QBpPlKEalraJQRA7vr\nw8ArSNUw8T9F4o9QTCLJz3Ypw7cSGxrGT3IqkPTDSasWYPzfelhWR/C2T4Pt\nzBwFjNq3SqSVZu72d6NJbOzJpfhQ1ZBksi3lb4MaUvEcB3+ltHLGvCkoqzYq\nxFSbAd1MWpNs/7mKkX0+59iVEDICBMKRNjWXsNKU4nLV+lEFxzLQTkRPoD9N\nsMP2axEAGIgidIUcKk2e2ExpRxK+SB6tydjD3i05tUEYpwFrnY07kiKYpz/s\nVSqjiCpMx3VcGJGHs2XyeSux2Uhq29hX0gDQQv+MTuLVK8OiI+8LJ9imiZG/\nxxEoBSTr+MIkdP+VZBE9cou/GzAG3C5d3eHNgLGt+jKZIub7JX1pmZW8Vp3u\nKAjyQRVU2pLPbuV7ldRieTMZKkTHlfpLa2fAGzoNUA+oWT9O/WxevDlEdeyL\ncV8K\r\n=ySUE\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20200923.7e94eb2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.3","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20200923.7e94eb2","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20200923.7e94eb2"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20200923.7e94eb2_1600844969023_0.5629873555081293","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201003.a485403":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201003.a485403","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201003.a485403","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"dff79a637a943c8b0e8c78c9f8d5c5c4edabd17f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201003.a485403.tgz","fileCount":42,"integrity":"sha512-6B9tcIYZ0vLqsDVJF4C7c1voxuH0qge2NcmidJoBxL/6VHhivGHdvNqiFx7qMIdswrwE8mFEvqZ5wXbehCXwAQ==","signatures":[{"sig":"MEUCIQD9sVx1yk0M/+Hq6HSKkjXKOdb7JpUAixyGMZrRrHv4FAIgaYIHRGZr/DLMctvnNSarGlOfG9Xxl3HYfh2pjBRDHiY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121556,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfeCM7CRA9TVsSAnZWagAA+rUP/iC4jzkvmtErlxG4v1AY\neG/GCNq6uHDlsl42tmZIjg6rjide4nNFvFxQeQoYGQ8RJU57MFOp/7WC4Feu\nOQVusAacUmCZw3qqGEHBYSOf+CshxUfw7oQENNBYWiHz4IR3HkhgqbmhmRPR\n7ggDa/Y4d52D1WkvhNk4393bFwwiLP8qvfS/O/gphlnVrA5MPzWirph4fkLv\nPPIZacmNH9T2zp4FVILwDwd7X70DsjoIelsBbE8xjT2xXusqCyYRToUuTNB9\nKQKokGd5FFkmUvOZiW44eotkQP8aOnMlNacyqwGsVyCzREe2DdivRU9aW1pF\nFK6+kPcsTqS2HEur23TVVw3OCa1c/qnNKqQ9vHR3W8FDSoBs+tEz+2mBTMt8\nqG5acxUkbIxQwXmf6PFS67mn+xfkaYzV1ZHNxMkYeAY24mVaXRIwrert8aKq\nRS/n7USlJZSNUV5kWbj531nsITD+nLEuNtvWAxpXvW345SwXAJakL+jxyPLu\nVaW/cafTOuRPEhNxaBJzFAjvUdAF+H9AjIPU/MDVYjn6tPbbS2MFA34WHbX8\n4GMNZXoxeCuoeNyB0t+OOjqL23v3JnxvKf3JVeL+G1CpZ769caKu+wKtFh2C\nj7XrySUnmeahGYMyGL9UuLWXcfAE9B91026BJrJH5WeJgzzFn5Brc+HFfks7\n6axm\r\n=9oqK\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201003.a485403.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201003.a485403","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201003.a485403"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201003.a485403_1601708858888_0.009354993030476155","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201006.29a8107":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201006.29a8107","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201006.29a8107","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"65d6b855560fbf627e409afb91e8535f1af7d297","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201006.29a8107.tgz","fileCount":42,"integrity":"sha512-6fe3KXib9CyTVyOcWmKqcN2awP1pYhQJV/7832uLhovV5XtVc6jXIfEQrWFxhPbNC+mQXGV/XUSiUgXkqNI8Qw==","signatures":[{"sig":"MEYCIQCrN88TLzzITaEPBtkoqLZw8kMPlfJqZrxF54kqOd7xewIhAJk00BFlCCIJel209ZgkSuoP3J44Q1T5w7veZDvOdboj","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121544,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJffBgDCRA9TVsSAnZWagAAwrQP/RtQk/OB3XLueK420TYd\n7Dz+o17ojXO3cZIOQyQHTkr8+arysiT8uJzi3J8z3SiES6nrDoX0aw09qPcF\n5hWJGCl1727DbzAF5ZtRyjj2cKCg1Nbanl8TOFCe/pSGUgvu2HN3290o41ms\nGjmfVCy4ycU86s4TWraoYnxf1kiKkjiII3xVNPsX0X0m0PjOKvag5Pg3oTmo\nzcU5hUmS2x+r/zMv5LmHvoesRn6dM+f8bTNKEKfB+aiIDyJ9E40ZSS6A0Otk\nIdZwPObAb3T13UQZo4h74YhCxHG+N6vgT6w9jZ4IFzUmnueqdWyntk0DzKdA\n5EY80Uu6EhhpR7OrmxbPvaJklzzu6pkE+IWyTDjCKGHFnHyVrCBBZQp8BPgX\nafxjzlZ8CYQXPwXnqZIbWE6krv7Qhnb5bxYBLWoZK0WnBoRCtix9tMWZk0FF\nqmNIErZb9BrWcdk+xt024fo6mdna8QqRYz3OFDp5mte4NquGL2B+BgxX5fT7\nRlI8sM9PGwoX0MyxOQ5eKP2r8s3EKY+yqBF+3ucY1uSQ6Tql7z95lVaLn0vV\nC8s2UkilvuwM2rJmvjt1eECq1LJBIsZc468U1bsOPzznPsGqUjgUz5XnMGV8\ne9LkoJb+WIUFpRD4OyIQShzVwx/igNNlF7/tKs+VaeuIIEvFOKI+vDwrt/aS\nMmIU\r\n=kNgL\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201006.29a8107.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201006.29a8107","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201006.29a8107"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201006.29a8107_1601968130568_0.46920184861324254","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201008.9bc8859":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201008.9bc8859","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201008.9bc8859","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e08527f6af0e0bdfcbb369741ee9b7c8b949a5eb","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201008.9bc8859.tgz","fileCount":42,"integrity":"sha512-utfKgixke7AWM1nhRxdAvYpHkIHRcZ7sWUu/ypJpwGDirPYDyLQDqk0vOD4lYyc3MtSazyp/6Dd90VaUzn+fEQ==","signatures":[{"sig":"MEQCIAmyMFpWrNjZ39tkDdBD4D6Q4ETmae7gTgiHlR73of2aAiAaaQGkzY/ir6vFIByupn5vmADVoyKX+k1fgByf0hjDDg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":121624,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJffrtJCRA9TVsSAnZWagAAZTMQAIGIK3PW0JZgVrKrjdzl\nSIpZShyZr7gSysdjetbc8JzfmRuRWak0mCRjuPP4QWKWKXF6uz3blYKmqdP6\nZh5pFuXle3+t9/Ie1De45xG/GCrPqr03vK3Lhxn6FXgPHr7grdl5SxIzkdnd\nlztuUID/9XeHfxtT5hulPy51ZEJbPmEpqCnyPTJW6z0ErkYH72UU6kiVKwj3\nyOOqeNvF0aCBC6traJSVXc+4SXJPHAzdqlulKs1KbglrcoLM905ESDOkB8Lz\nDhDvmYGv8ozumtg540W40+zAasYOSE1HQzyoCEHAFrMUrlNh6nyB3CYH4PNq\nnlaGQwqjbLFWhOh7IbF+WdbcbQMpf3FYs25AzOym/IEWsNRgvUHxPmOKfZCX\nZBLpJ8BdVe1P9JDB67JPE2bXxaDHo+AHd0oh9jpwDRbeClDRe1KQvDkrYc8Y\npW9BG3ASDwWjhhu3aMG3hSsfwBICzinLblgzWjVVL+UTSuiDcsJ9idbr0HYF\n4J5UXgKkLdgIlLabJOCKivqKFaujpx0s4NvUdvvn+4bM7q19RvcP5XUocssV\nHfNdRhtbAC45AYFjek17aF3oIOdoBoUTzD1p4l//WUizc9RtPt0tRuCbhS8l\ngipi++mET8cx69w9Al8iWVrZHX3Kk721Xh0Liex6jjKGuP/3iIOqCezkw9x4\nYfOi\r\n=v4sI\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201008.9bc8859.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201008.9bc8859","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201008.9bc8859"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201008.9bc8859_1602141000753_0.14782314418370213","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201009.5113321":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201009.5113321","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201009.5113321","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"4aaccf73b19da76c67136cfbf9af89762efe20f7","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201009.5113321.tgz","fileCount":42,"integrity":"sha512-iIcZefNhZ/lh/RPYDpduIM+y1TBXx2iQXpIBBAVGXkSe6IRBmg8pNLmS2KmQj0bBzc/5hdK1TmEcAOSKzHHmOQ==","signatures":[{"sig":"MEUCIQCpQHmVXXJ9nutmU+g2SEG7OFV4oxmeZy8GcjzLmlannQIgbsKGdIfpr/BFFrZy61/G5yI8fHBxRfzlLc81siEutHk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":122345,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgAyaCRA9TVsSAnZWagAAMM8P/0CBAj0ea1pFTj3kvxq3\n3w64UIkdNA8B+rfouGDgC6NvM6MNLJ40JbANhhyypt5i5bKtWTgqo6jXrQt4\nTY1kMR9SbG0BjY25Josv72gl3yTuDZx4gYti0I00vdGgZDZhOsFR6fq6V3nH\nUR4cjo4LCHCsRuDQ8naaHCikpBufboiLVs/NghMujIEoonGIsv6BUkptWL2g\nq8kovhxdK8iUDBNA41On/pKlJ4WBl8s5YNhYDl9CAmDwmdickmVO4LsDxhVC\n7eYbJ97FtIuFF+gXovRjtQeOu9WTp5ILrMYZmmQAPOogeM4pCYIJV2EBZGi6\nfcfCeIcGzgVHGUMuzvdWF/kXKG3J7FtFJMnc3qKO6Pyq7/ZEIZF2PaRKi4k1\numHrSqMH3uAcw0WTUKtXiXL6wdEuGOLCPCHbkALCzvQtSycUBR+kcQMf1J2X\ntyNB/S+nPjeomfVoqn6MO9R2y0pbRB/QL7hUJcF9x7wTwLYVfdutf8X8FkyL\n0R/SVSEdlcq/91hNB1xgYeiQT2zk2x6ko7jf0FVJbDyKty9BbyMs26UzujQJ\nTQFfzQAW2jZjfuznVv6Q8ZTfFGQszJPlLCsmI42nJCyzWarcCash66Cr4Mba\nhX3kPf1hRgloAhxBvVz1S0Z4xgu3ZTe/07ehqStUnJr3FgRqUzj07v5ral6k\nxTFh\r\n=+kVZ\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201009.5113321.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201009.5113321","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201009.5113321"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201009.5113321_1602227353659_0.5951665368130641","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201010.6819330":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201010.6819330","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201010.6819330","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"8e91beda8383e5859340fb48facc50771eb445ef","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201010.6819330.tgz","fileCount":42,"integrity":"sha512-CKLzO1WXLL6hs7dROAhtqpDMnivO/7/CuEdARo2iBOS89Q1mKpYGa1TvW/H81WzfzTHF/7U926CqK4sgyLoAeA==","signatures":[{"sig":"MEUCIQDHRZmYVt6v8E5A/mTLSmxlcF5336tl/EGLVv8WgUjPxAIgZlJ34eXwQ2f41EcEoH3jexJxE3Xopi3M++jZy5h7ZM4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":122319,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgV5eCRA9TVsSAnZWagAAdMgQAI8ftVwGKiv8aPMvOfZz\nvWAFiMdSNlzWq4rHmCrqpOaXuY7//YVCU7PZZ/CK9W+fZZML1ErwHecgnWbL\nxo7B/+XE+LUiIMn6l+bABLOQUcG+5iNN3sasXjp5uFpHsP81P5JXi0S9Zd2c\nwu1QgHna108CUktZecm7s1NMbpMyBSeBb3ZT7kWo65Or2XJdLIERL4rj+Ldz\nJMc0N4TJyqXTTXShC6pOtFzq5s4dfdnDEiaW00gOzv1DJU4I6stMzVNMg5Ce\ndMKyI7LgJ9i8/umXVzp06bDh0IsjbjjqYKTgpKX6Fso7QvL44uDgqKM769Qu\nyi+eYvDXNSjSh1GCSFDUfNrodSC6LjhBDuEOdJ78p4YhQAnC6apBoGA0ctYA\nhRlWdYMSOj9izmRHNP5FZm2h0s6rvLwGGh+LA3694EY//ZR5INz0PuzC422e\nkidFDYp7pwXMgpNTKoWmF+d+fwGpgs4985DJTDE9AdgNXd1tgCYEL1xGv1uN\nBflJ1atNfKhi9Rc0nX8lozml624Uwt1gGChG5JAx638o1iNI42z2XwKazzR3\nHaiMr+dGI8nHaJR4dSEhfinxwetLXyofHoTyxwQo50RB+OycNda6T2FiSGkJ\niAef9FsHHT78ed5QGJrlmrj9fWwuoZowR6TO0kM1WcNpT7s53x9Po00SC2Gb\n4xUD\r\n=Tkdn\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201010.6819330.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201010.6819330","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201010.6819330"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201010.6819330_1602313822048_0.2556437373836369","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201011.d140767":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201011.d140767","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201011.d140767","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5205bad1b237e0d33fe3397f0a681bf6af7f3856","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201011.d140767.tgz","fileCount":42,"integrity":"sha512-5euN14vD6v10BfyV4cU2cj6WNmzxzgjb1mVRODdYoVb1+hY2hC88KzRKq2pTyQbQOL5pSVPbz/ZDxMZe64BmDQ==","signatures":[{"sig":"MEUCIQCsvM2xr+mraGD6BK/oo0XmYnn0qlgnGkq88RaZfKgcxAIgJ/oAFigI/LNx8YNiuulJv4hiITUw6kkwXWG2dJy9Wjk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124744,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfgq+aCRA9TVsSAnZWagAAN94QAJfj0yJ2fR8azOwf+j4S\n3GcWtAK40yfiwMIlT5NRZ/yyXSAjsiWOBDlv8cGIaX1IhsHzyYzD7Us9bwNQ\nt+j1lz4C+yWIXmxfUIIv8+smf6ZOiZICLphYZruuTb/4SojymhIB9w/CF2cs\nZcGE8ZjpIeVHR0qGc5dna0ZpBY3Vf3j756DxKSQyjkj7DBEBSy2dx8JQKC9M\n62etLa/8J8D+Sti19KGeXvK+fM2trBr/9nz5BBmGKp2h+NbMXvtTNDmrwuGZ\ndqetRzGJSi1B73fVyILDbb0Jxmw2mkCSDHJFeSHxw4af4C2N6ogW7JFgh9U0\n/TG5iairyalwKv/ojLwX6w3zzV9ifGhnaycHl6UXgkEZ5Pl2ldWXUnzjBC6K\n6NPFpXQRsUbCXfCsH4qGMrjPWCVlnCEL3FXOKmY/Ub0rlwCOHIQ+yIFuNNcA\nZnS4I4d2KhRLaYu44ENP7RO3njzw0SzeeTkwA9dBN2KROnKna1kgAPAWJ4wz\nbWerKh2IOSgZj4YClLYl3s+j9Tlw09UhBHzO/Jr4hruxFhfs+lpa2Z6IN3Ho\nfgW2dhlXBYU2QSUbydIn3mlaBlHjQZJw5l7+NpzyDh8qftZhDW9qEgrlLQQ4\nLAo4esJfWo0LT7cUe5jB9u2tCCZ+cFtbtbRcrzBw83CJg22r4k6XnekzZbdo\ndO0f\r\n=7a+z\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201011.d140767.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201011.d140767","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201011.d140767"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201011.d140767_1602400154092_0.8609292982885601","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201013.20d7917":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201013.20d7917","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201013.20d7917","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"de128ab65a8536893f38fac7655a922e780809f0","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201013.20d7917.tgz","fileCount":42,"integrity":"sha512-lU7qKQGmyhJw+w3ZW87oXXYzLuv3H7ku0nvopQmHsft0uCYv8Ap5Y+PaFJdBhhTQDMYO9mD+M6P/E39sMd0b3Q==","signatures":[{"sig":"MEQCIFKeKmIkuC8/5FPeE+2YQylfsnjgPoTDm+8b88xwV2h+AiAcKYx4NZqHdBBqRY/n1B6t8IDDnZKkf3Gt5BOvHGy4mw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126488,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfhVWbCRA9TVsSAnZWagAATKUP/3TQK7rUlEY5Yp5gHe7K\nG2Hszj11vrV6GGXKnGvK60pAO5szS5wwf6Bb6u7tB2s3gwZtLyKd9944zlyf\nHxTBdWtqIOtInyVnuo6rZ5EC+phpsAQoWIKrdYgFLtyIp8YpQviF6tcjRDS6\nxdHD5W/axpbhIznhsiQg6qgq+detTG+gpvVgkBZY1+ylpmJFqm2tQlA13aDh\na5YVWYQkDJNkDI4cJ4XpF+bRXKiFOixgJLv12sc/9G+Lon4yZMs1Bqi95VJ4\n7gu2v1HObhY5QpzgEF03SWbr0cEYVTPLfoxNJ2ZLxOtFOqJ9KRLiwiqWtpYS\nTdKlyhKa1+BicfhxCBESh4/hzVSqbq9tW70TtZaRpa8/fuL7fXkEIY05BZDV\no1QDB5b2TaxXmDnqVDfKsQjAzaJiwgX5ZTTKbhm3tONlWaM1r3ioHcm+laD0\nz7yQKH2D+/7siodRt4h9w+V8hoVOIcp4/mkyfS20uY3J8jKVw5XYUZY/tK+R\nxIk76yjRc0JfGEmFEV0yfatm9FiEL6RV7TWgTKNox7rhXCPDp0KuJjVZoMH4\nR5DhUoZYLKXz+CXdNiXZqGm7NpTVsGSGItjwxICIbQG6pwlnF8e8rLd9KEdQ\n95dAfETAfdYDUXvNFPtXhAJEbSU0bvwHPctoyNWY19K676Z8V8F0nf75r2sQ\n65qe\r\n=DERT\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201013.20d7917.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201013.20d7917","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201013.20d7917"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201013.20d7917_1602573722682_0.11228314522289673","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201014.ce6aa6c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201014.ce6aa6c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201014.ce6aa6c","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bfb4c1cfa3c4cb4476f8925aad7f4c928dda7353","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201014.ce6aa6c.tgz","fileCount":42,"integrity":"sha512-l/pT0E6kmsnQDIEB/yIcVNm5reE0WrFVFuKWMJjvOMIsSI30EtIicivpZgOXYUrvWTJM6CN0UjhegbeQDwgc0Q==","signatures":[{"sig":"MEYCIQDmuwb/pEbq9sKB0Va09Te1Gxr1y2gvZBa80ePGr0jgAQIhALWAEnjkl3n97o0kXYLaGLazXyXdvY1pZj/SXA6Bnp2a","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":127624,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfhqSRCRA9TVsSAnZWagAAY/MP/2mOzR9INQJi+DtECmGo\nAHjJ1X2gYk1Frx1sB70rt0J8Tacnc6kZJ3N4kWinbTQoADTO0fYQrCz72BHN\nhI4sICNFKd0OQjZxk18uwUrALv+y1uFz8zOGAXXK+YDDyWrkNR1tx5W7Fkvz\n84pf1Bwu4ozC2exdPv48HG0RlgT14aATk2eYaF2RO/EqMV/M8EOG7mqe2Fz/\nvv17EnkA8AGx84YIhMorEwHx2kmPVaAd8xUYP/IbVjtnDpB0gcNlqeUj8sek\nWdIhRpx/n5+p0Rz8VOL/zim9Ucd8w/OyyMBWUWOW+etW9BpzmAIMLe60Abbf\nwUXVr2UGuaOSxIMfrY15CX9l7uZI580OnDryPXT+36CLHc0mJuhAgaa4q1YX\n9/FLhJ8bYGN2pYYqdJGlSClOvp2wNpuBB9yT/ZKpjdu3SXH6k/lAwkV0vRCP\nfjA8wVayd5HGcN5x6AkGkmloNvvxupPKG62HLhd7ShEBSbS1gRLPQU6bQnbt\nHLM7Lidq+blmQMmeZl4N1F6SVZHOm84N4KgRNaqhUzHXy79dxwJcuFOadVRM\n1s4k0aYdj+J/p6qETCYtyv1yLC3kHdSN1HxlJL0IsPvrStw8Q2wYSUDUTXTW\nyexPD5BgV/bzJGvlVKU852oDgbA/uYTDPpHosSHgduGs7HYmp/tZ3yBFmUb6\nbI/+\r\n=GOc7\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201014.ce6aa6c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201014.ce6aa6c","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201014.ce6aa6c"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201014.ce6aa6c_1602659472585_0.09155461670946896","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201015.008a3a4":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201015.008a3a4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201015.008a3a4","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"9e4dac67b3e3f3497f79e65661d8505662429dc2","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201015.008a3a4.tgz","fileCount":42,"integrity":"sha512-Wqox4rwPZSoCjlj5w+StbKp/i2ZXRlY2NkFJB4946hAFA/V8yMUjJcs7fOU0k2W7hD3OaB9YVptMbK5w+6mlOA==","signatures":[{"sig":"MEUCIEbv2Z/ZSNKWZSbgbt0FFklfAeF59wjyesNM4VOevTR6AiEAqRCBzipjB44afsOmJRWo7rr4q64X8ar+mJc5GWCaBRQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":127624,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfh/eNCRA9TVsSAnZWagAAwOkP/3wD574OmxUT53BXmbeV\nwfcuFbr7Ip19OJu6QOpFArs1IB4SWCT2jrMUNqHqwUk5/7Xoy8LrCu8cn4b3\n3d3lEceyUu5dkLV6bYFHWjDocs8uG7NUL6XT3oPtmqsXhtsABRBBsqDIGnqx\nhPOZeWelGTtFkEpVugktzK/7Uq8Bio77to84k/gqdBBz5uX8rNLZmQFtagPm\njeOHS8Uf3ep1mqwPnVeNvygtZMrlUqB4x89tDCFKZ7fAZ6ElmdvnS3WOwnLF\n+xFBPk7eO+bLDI3qyh+70rbVYe9kxyl7jU2z86Egjy7nyyRJdlYRSRFyjh3c\n9pGXdFc8mj1PBSoIbV1n1t5MnQz2fnr53nUxyqYZp3SYaO/yMIaZFHrYF3S+\n7iAp2gV0KJ1VAKVZkGT7ww5iYtlCH3wqr8sQgP4C45SYaTb1EUAYvy5Yh4Lg\nnBi8q7CWbyILTtTivBdMAgaD1mwHke+TMhe16lrSA6ZAvUdgAet794mi3wKL\nbRjP5aHQZ5dVdHeyRJvCopMFxaAlMZQUyCrONVRHMMgB9JzKTMwUHMmtb54F\nfo8DUGMe2HCqFGNnLHUJ7sgoeHqtPfYWNCnCxw9aY/AgOT5LN8KV5ErQUxah\nt5tH3f97+RvCYzmKJTb0BHK5/g6IqbwjSMhdKW6o3HWEG5j+lKueDQnDXgUG\nMsPB\r\n=JDxU\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201015.008a3a4.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201015.008a3a4","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201015.008a3a4"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201015.008a3a4_1602746253399_0.9410595521195402","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201016.393c6b2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201016.393c6b2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201016.393c6b2","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"af9ef19c6d8bbf3a9eb044b0455353f45b7b775a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201016.393c6b2.tgz","fileCount":42,"integrity":"sha512-AZiAnJJ1hrjm/CRTzYYitzbKmFXhw9ofUo+1xTu+RYPibIGbaOOjoaPkeMdTFPrb6MfC8d94JIHR9M4D+xgnkw==","signatures":[{"sig":"MEUCIBp3ibShElNMNInSp0stAp2qWV1y98zbNEqCd9zkfwLQAiEAkXjAhppc0xG/W0vJMttZP3uVT2Ea91ps+Zb1MHHzCgc=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":127864,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfiUk9CRA9TVsSAnZWagAAjOMP/34P8AWKfTBVMo38XY8P\nrOtpAYr7uj5PGtAP0XscYnYhm/GKxrqI43eZz4F4xQxUMjrQELnYWRnMbAQl\n7F9m28mlVA97+E2/guFgIHlUXwEWXIt/73WCtG/i4ZT1cTta3XOjxiqGx+kA\nX0SnCFmW/RqUvp0m2Z6bJIwtNlZntqpZVMWrNzR2G4RRqlsTLPky+eaoZKwL\nC2TPQgjPHk0KEkZfBD+hZKUfUEmEwkwKTn/Ub31l/dJ15mURM8gW1wBHmqp4\nrMCnccWChL764+xZYjw9s1bvwgktY8zwKHzMKVHTVicF9iCkLVpT7oEXR4nQ\nUcoHLm+tb00MJkGi9cWcRTZgyNy8ChmUYEFBDhTGUCj6tybV/WmJVH5n7MA0\n2thevv06PRjmM+hwNf1bfZ+vxTvxCJ7ivujYJDe5X/oG5jhrRxTIxoIcrztU\nAiToDi8beysfHU3TLwD1oVUJQLlgPsHs/+MQsufrN9WgARJhm/hmtRWn6q/M\nJYBmjEpt1Cr+c0LGG1WBiaHW1Nhu4Ft58hNlmTICpidNqYM6xFfhUzFY3PE9\nzsq5RcTWhmPc3miZkI8QHOe6o3Ox1xa+EnJckSvcPJvQblt9QH2LcpzupcQO\nd6qVswJEi72JkOoa7/X4GJWtrbDKEGLKIg146fPJdnnyDrpPiMpkGKwmWAOZ\nbT9a\r\n=O3ze\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201016.393c6b2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.6","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.18.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201016.393c6b2","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201016.393c6b2"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201016.393c6b2_1602832700634_0.024892704064004567","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201022.4350e73":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201022.4350e73","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201022.4350e73","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"89a96453e2f0de64e9665961d35e6a458584aa17","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201022.4350e73.tgz","fileCount":48,"integrity":"sha512-gKTRjjhYcgITRDIoyK40YWT47YVHg6gdQPhMPao2/jocRc6q314Lx2h62lpEZDGfQ8/5zOJrlha4TzZP9oweoA==","signatures":[{"sig":"MEUCIG5azQSKdkO3IqRsVbNsQQJG7iTZlzWLveC1H1dKOaTNAiEAtuuDKiIGYrSr8cpxu/r5MoTgSfDN/oQrGu02ZSOAxKE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":145929,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfkS/1CRA9TVsSAnZWagAAwNIP/iieXj6yUliWbB9huXLh\nWQ8eZdtNL8nTuVozuxtEb4CUf7innEFYCJZhG3rZCeSRwkpnRRhz3Zktu2Tl\naNcQhEajuAxwzh8/37siXGmMTIUSOj9WppwNo05pIqi8uBNd6pVBNcQocelu\nvaawAQmKDE81SZyA8FTJY2XDVPzACsnp942D5WEI+3AmLhvzom1hbSXuzi4w\nVRqIVIlH8T44iPjY3h9MbBbvfn8WjtZtmIb9yVltdV8FMk/gz8t/9Tnkd+aA\n23VO4li8LX95/j4rteiQuk3mwjyQ+YGabbM6OKmUBSiIt+wQsWsfG01Lqe/7\nsEjGGymkVp2vihpHvVhGqvvR81hxwDXsRnQ7qUtDVGQRvbItqYuIWFbWzQjr\nBayYA5jsXO9OmWjqqCv2cjxh68jqs6jUeXKTtDBz0+ngJUp+F5Wvef7jMydc\nOpkBCe7y3V6dORTPJ9vRy2BpdxHtj1JWqHfA4X6CC9DUgDdXLh9KHAEJLKA2\nsFAFTL0p5DbiB39ejpuRg/28L+brK+KrKpF4auquAF4227NQqzwM2h5NUKLP\nDpW4MgK0KMTA3Do4wJYd0PmfbUsG1W3h/0AVlpJuMMCDPBl3CN2dAaK5A3YC\nhj7mh7WSue8eW6zbip943Dre//QSqLUFWUBDAQ3NBIRViCBEWQSreQKn1wkl\nCDCD\r\n=jIPS\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201022.4350e73.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201022.4350e73","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201022.4350e73"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201022.4350e73_1603350516834_0.6250876327408863","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201023.67f9a0d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201023.67f9a0d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201023.67f9a0d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"aa7ecaee493bc3a110363ee060106244da74b755","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201023.67f9a0d.tgz","fileCount":42,"integrity":"sha512-SqpPLoCZxNs2tqEnZnmrRAHqDuHOjq+a8g5TtFzpQ9KHGNfPkfjbQc/gCExKEdMIgwucVR6/KJBaFgYuItTnjg==","signatures":[{"sig":"MEUCIEJsNkwkT3itAQAExbz2GD+uCX1COuFXGg5Cp3jLXBULAiEAuPslLRu0/jDpP1tSrr5fgEul45aWPEA2QzYB+MfqNWI=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":138721,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfkoHWCRA9TVsSAnZWagAAz9IP/24cq3JqCwUrV7cuJEKc\nbiZPQXFrnp0DIXDpN0rA5jE6tY09ygmwt3h0B9T5zHuB4s39wMT6mm3SlsCm\nb4Qp852OoAm+CY9ACsQn9Si/t9uW0wBMHDw6587NDALw4e9eJXWV5ze21Hkw\nZAwFKB3By33KVqGHzuTMIa5Ny1DVrr6lYu120mn15HRNsfNpbBouFwk+g8GF\neIJfNtldHpC7PXR66FdMm/CAMSdqdDQ7u4tpNSjjzA55SKLGHnOJxW4cjuXr\nCcfDUdqMtiB6Rox2hM3UDn/Ccb98YpgiCpAAG0HlJsuiX6abgHpKkwnnEX03\njdhBWQTshnFiTjZ82NexJak9pEASWk8RwzWDzMuAvLcGBfzysUwQ2lAUZ4nN\n3L9eCMD9nTE+CMd7fiBoF2QAVc2qJCRV/SNrLj3Ei3br1viWosbL6JsJXPx0\neyfXSvVw4SjSlDC6Qs05y1sJjEkqvCnw0XOZFRghpIXo47SqoEfYHIDxpNML\nenPY4AsTM+AGUQektjeA64NoYR9SjUba51/HceZt1dLipS2pNWwoVHswynZQ\nkzokkKhAogoGTU8R/PdZntusI0E6vG+I7ly0KDeaurvoO/iw/XBZ+ow72ywV\nmW10C+v9QG5jFPUWw9IRBLQf18LqxaF4r9AtLgcYgjUfmsEgHsugNXXOWiuX\nBiWC\r\n=M7jd\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201023.67f9a0d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201023.67f9a0d","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201023.67f9a0d"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201023.67f9a0d_1603437014053_0.5753335596011238","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201024.23949ae":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201024.23949ae","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201024.23949ae","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f50b7c4f12ec4f60e85294fca0a553d8a07a602e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201024.23949ae.tgz","fileCount":42,"integrity":"sha512-uihRniBgH5iF7z0+MtnlJYasDelZ2+gAxHZd7pDWUoDuKGy+3Ypbz/KkXQHX7vgIjr8lcM+N+8Z8XZE764IsZA==","signatures":[{"sig":"MEUCIDrD4SEmvZO3urqM0/UxsItXe+MRGsjZ0j9fOuxE4/5QAiEAjh/h8/b7sC8Xgoc+IlljahPBUfntTo3O/SFzHLVo1/U=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":139030,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfk9MRCRA9TVsSAnZWagAAIl8QAIPSalxBtLOXiU02Yhjj\nvU3zno9dBrqiCOUmLOvpBeKh1PrEXogXuiwwpeXWRV7aABfKaaPbqS1ef7DC\noOm6sJ83ZrHycDIS3Z9G6491NnPXf92ORFJLMDWNkhN++gLEzBtnkc3nvBou\n0ymg52xxJuvSgLcdhccgALFFPmhblpghAoLJhaye5KrCkdaeuoKDoi6fgClb\nrJe55PZQeT7kugrwAq3NbnNHh3ep7YJcZ4fQJd7epVEuR+5QddduSYnX9Crt\nrmDG6qOBG95jRFXi9B6m+QzGmpom0X9QF1eP6ZpZkn54cu+di4w+Ewhj6cQm\nh33t5GtAJuOjHh0ja/ggYbfYcT0cPCUfcuHv3F1pj4uh33swm26z+zpHcq6Y\n7ZSpR4UbeW+zj93j8zUFz1XQLjwIS57YOC0hSxOD9xSO9Mtcs6vDeZhmxLiQ\nlrA362Aj4sCOYm9Pw4k6jX0Vm0SmZrgMWJcKc5iExcS2c2LB+0lIx+Hh7THo\nxlZh5iZw5tMjyf91468V5lq9LBdpT75w+0KsNKUua5omfAgqWunrkHoUJenE\nMj40J/xQ97J3tMQq07e+VlAlqRBXiTj1Kp1kdb9jFYvSY6qeEWX8zvo1lwhe\n2Tb369U0AmMG8SWh9ENT7i0awRfqs5Vy7wdrUCo3fTtCVpx1AbkcRnteqJfv\n1y9j\r\n=tbDg\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201024.23949ae.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201024.23949ae","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201024.23949ae"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201024.23949ae_1603523344987_0.8120872298581954","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201025.035cdcb":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201025.035cdcb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201025.035cdcb","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"98712a1a88f503a3ee187a827c3def677bc8a1e6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201025.035cdcb.tgz","fileCount":42,"integrity":"sha512-iOtoJ8G/DM8uttgbSvAfYs1yfJnKjC3sWSULHMQT5xXNOS6Yxo5YFvKYyw9IYkniSelKacza3cLnJxZQ/SPSCg==","signatures":[{"sig":"MEYCIQD8fTkksmrijgABxZA7SLyprnmRpfB8oHQ/9FsOYY3INQIhAN0ZxVPc3ti716j++2+sIaxPyqvW+UkVfMen5WPrXMr0","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":139030,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJflSSoCRA9TVsSAnZWagAAwFcP/iopmH4BBojB4VpW2FQr\niXwYIhF3X7PZcQbiiI/ppYT93vTOu7SDPxDAr+6iqEnziXDZ46Nb9TWb71et\nMR3f9+uMMv2JUQSDQNJiQbgkuyWXDJiG/3uXhlzmMdQdujTdJy08AtKlk8ZP\nSThqzNzvSxxC1+OwymMQP0RjjjGlnyEgd9ycKbG7xlpB1AumtImYUa/WFRwP\npsw3+X3wIIfhZee1qT1TEzynK/djWEtiNHCbbYFi4Ps82KahbE1956pn4hsV\n+FzwrNULsigI4rOFHaROt+2pnd+BGWxgEBhmHqLtONMDDYyr0o/QU1R2bkkN\ntkvEhjh2wieVMHTyJEwBw+MmFzZA4s+F/OE5qzaA23D/M51+7dl8ofY4hcRg\nnc8RemjNwPbHJI7CznaGCXsstyt21QFTc3jbfWvkBpK9a6qYsazw0h84TnOv\nSGyo53oxb7oG2NDXx/ISjLXbTz6PgHwElU/bWEb7klmdYFqW0xdWVBxliGRy\nO3CbsfnyFyOV4Zjxps2J14EJ5H1V8U1S438kiyjFilbrCg4QWscPFHXgRvfX\nH16A6RJw1bG1svdh/A8AQdmhN0BW5U7+NRDlbCu/rdKS83ff+Z7ytTvRLZO8\ne7JpI35lRLgR5LzSJijPDhezkysQtOLo7WtzSrwPtrWsfEarJL2weqjaAxNu\n8VgK\r\n=Dldj\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201025.035cdcb.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201025.035cdcb","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201025.035cdcb"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201025.035cdcb_1603609768413_0.9906960432312124","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201026.d8a7aef":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201026.d8a7aef","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201026.d8a7aef","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"97e6a633ac540a13e9119e17957506eece934049","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201026.d8a7aef.tgz","fileCount":42,"integrity":"sha512-O3tbUXExAR4MoHh9zyEaSTpUkIq8EjIp4CSyZ1qKPpQFgR7osUOBeHqTT9HjBkN8MZRbPpXFhqknNpZrQrvu0Q==","signatures":[{"sig":"MEUCIDpm3G/mRsT1gsg7B93neYigXVfQjnBLSJJGGXG0lBujAiEAk+Eq3ZRTicFjLBUmmYZt6nioiqZillcg36NBin9sHb0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":145003,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJflnYxCRA9TVsSAnZWagAAX7IP/j6G82vsku6jkH7gA+Qz\nDmuIYdnElmYqKeF3JD60D7VJbT9+uTr5hrXwSyclaVK4AIYNGm1kB/7IJ9oF\n6vwjAyfEHA47b1ccXi3HiyK6W6r2KZPYDUS2GmZ6jhctX6THEQ5c/uGLEWIv\n44Pl6CLuiw80GCvlblMueaexuBLr7BShw0RwDQZJRQfHNJgmhYCuirexP1jt\nQaJnobzoKHMgvzD7deMXu436zMeC5qdw+Jr2WeKSV3pK5xJ2+ArlfhQWvF00\nlaz1phmfrQaRettyDZIGCXyZHwFKx/dNWmBPuMYVGpTsB7csAzHT8YfPuXLA\nZ8fhpw987lRiKqYtOx3qoed9QXZ+6IjZ1mi2JBVNmuL0qrbb6ADFwa6zfyPe\nJqez6yYTQFsXc9FTFVw2JcTXGXM4z6nBQveuRXNrsIiKYS9HT3A7fiJR6+/q\nInHoZfQ5dWzCpMkjVKd9jezJ07YVGW6ZdBJ4eammTzsMi9EHrKhtD9mwSUiE\nn+JZcj84BpebUA12nP+l+74rmznCtq4iOTC31MZ3LOqth5zi9UVZsMCPLlvF\nusjmrSKPimeqGY7aqNAPM3ztPb5159hFLo5bA6ol1sqgJxj9FqkM6AvCIueZ\nBKTBQPvlKDTw93lvOBOJrC7WevTxhqgg/uaSjkHHtN5XN0Z4++lfbPqMjIAk\n7+2B\r\n=ujFb\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201026.d8a7aef.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201026.d8a7aef","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201026.d8a7aef"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201026.d8a7aef_1603696177126_0.21149062537400498","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201027.744fa02":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201027.744fa02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201027.744fa02","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"a13742210de17e64b5da5d6210ee319c1d0d699a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201027.744fa02.tgz","fileCount":42,"integrity":"sha512-wEvheH5j5wvXKpzGvoiGoClRal+Ij2A9uEGbIT/MU6xnx5q9rmnC6YMDXJqolKUAYCe9zGFf21v06DgUW571aw==","signatures":[{"sig":"MEQCIGf39ixZfZMoZfJZ4ufMEJzIvE465tsn9CeBPwiFLRfEAiADDGsK7YP9HJIowvMDWKkS3SnK80aUTKCgTDjDs1BiOA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":145003,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfl8e3CRA9TVsSAnZWagAAOzQP/1WgGLhun5INlenUcZJN\n3UDDFLGsFCOFkiwWZHzr5YPzY1xCNAlJkc2fe+pWGKC1pw8/CX0qSKsQ3Gfm\nP13+CuoJrdwOn6S8j3tOA43GkAcJyS8eHGMN+sklTjR0VGPVMUR46/Mkbi7N\npSUCSdqsBaWOmk0yveqzFC7HH1PTZYqsB77Svl1lmi0kTRj6WimSEJBHQZzI\nB9HXqArmAVslzYSQvHx9zWhdE+KPTG62LYaXAaUg3TMDk4P2zLoggXa9bl7S\nquZVJFysMZ1xOXCMUboLSckbax5tdGRGDj1FEg/UnRMvoWlF3sBJkl5rBjn2\nvy5j4LNyWq2YRLdynua5FkPxl/sZ8DII3IVsPHtOLIF24WDGlHZc94HSc+Pm\nD7GPa87mXX6WFy31JDzcjlUEQlhAwmoVrDo7Ae5yMRfonqz1CyQUsP57s4Tg\nUk035d8wfOozBxL41ORRdSliX3vvFEp3fwUxEV52Nlq0BHONvQk/LBrBNiM1\nfOT1/ZdqvK8/o6TKgQUD5iXYGcQCXYV/Yy6fVaUbQP6iX5b9K7FJnAIqQ0/9\nlNcCIRc7IZoPPFPnKafeIBy06HGkJksL3y/n3xarjhLeKsOvm8E/sqtnaiky\n3JNCJpZK/lxE488K16QWS9YJAdQEZhpTR8j3q1ygX/ss2NU9YtSxGp0ulxBC\nHwk4\r\n=EbIG\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201027.744fa02.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201027.744fa02","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201027.744fa02"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201027.744fa02_1603782583475_0.08316390065309598","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201028.1b0cde3":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201028.1b0cde3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201028.1b0cde3","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"385e1dc389c20b2258e6ecb1cfa4f29f9e989c16","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201028.1b0cde3.tgz","fileCount":42,"integrity":"sha512-oyh4PJB4qZlMtMToZ13LfQFP0ZSYgPrpmCh37M0YVACfk686xf8OCMXXGzaPm5f+0Zb7RQuxcUtLOIkvVkPM0g==","signatures":[{"sig":"MEUCIQDDEIbzVSoFHkQxIKDtaLvIrRxa2f3wy+TOKp3UnhrbhgIge3W+NkfivDcfS1A8OxLHHoYsTs2NwIXR5PRjdK9R+dw=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":145568,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfmRk3CRA9TVsSAnZWagAAcmUP/jsAjExkLNWlLXhk2/97\nbrh2QK9Gb0Tl3bXeTQiyytrsr9szweZ75xK97hA6j5b5TJ6VrHkSjiPCoAow\nPF/8ckMq/39iheFKdolmdm8clTV8/KO+C9MyBFv7wtNIM96GQ/mWV9SWdw3S\nffCS4B0QGj+9wFcRyDNWZbvHKf1/g0JIKEJAxVacbT4bJlv3KMGuBRhbdqQu\nhcSVfLgZmDfZkHNZ7TPiFajsXuRctxUTKRDVyibuFeg43g19IwzdS+5PStMI\nzXhY1z2idONbh31sGKmqVeAkbZDsYQTXgk17mzxbvYewNrtWvExqmm8+9bsm\nyw7mpWrRQpgvh1wDgA70cJoWaqC9Ycl3+UNDCDAhDFD2U/x91NEIIAGVLieT\n4y2Ruvr3Dh7i/sbus0A9PCQTEKo90H4KLov9me9Qv92wPqIhjGD0Za15FKCy\nBIwZXIou3VrSLmsUZWMs8fNkU8sZzdgXmP2mVxaKRSWL8e+wKsUBf6J8+duO\n4cwUVnasX/TMrERdnx6Kk6AhIs28YkkghHqNu76lF4KYANn0wolqUjRnr7CF\npabmKOc8fwwqJJJmaGOD//+ktj8NPd6CkrdElcmDq5OstwBFvqRmMye0WpEv\nMifmFsKjVei1XpDTDt+aPkrFWo4H8tmuG49su6IIq+4Y2vXDKp0XEIg5tWkc\npF5k\r\n=mcCG\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201028.1b0cde3.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201028.1b0cde3","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201028.1b0cde3"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201028.1b0cde3_1603868982867_0.13579881176414044","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201029.e864bc8":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201029.e864bc8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201029.e864bc8","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d433cb0d34a8689a8dde16817552593e8ee6023c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201029.e864bc8.tgz","fileCount":42,"integrity":"sha512-tG6Nw1+UJGqjbTsld1mq+LpKjXXDPEdfSGwy485k2TQXXkaOAd+SYahVF/jHbZYclRjkLS9ztCFFI8hwWfFMaA==","signatures":[{"sig":"MEUCIGRr1JH9Mt+eSP9Pp36wVI2eQGvtGa3swgh3mAh9TMQDAiEAjKuNFg/ZIxE0THMLa/XgtOiad0L+EhZnQPuRlxIDYIg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":152558,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfmmqNCRA9TVsSAnZWagAAiqgQAJ5VzCjjsYITn5xTtajD\nSNucRseKEO0gThfK8RNbgQfK4WAOO0WgHoEkLl2M53ukNxgelk1FvaHCFxR2\nGjZnZMLHqCNYNrUpixHirsjF9pTZIRBnyn6Xg0RemjVv0mFP2ZW/ApGNEQo5\n6u/XBvBt1vQgZAXjr6wfyuOotBmreFaEYTifGWbSv0uhOvaNSGKOd/TBFCr/\npJfl5UeEfkOZ9uRrWR+cRE2j6Bh9O+Cr8OTT24r9UNXsnNLPPUGRzYB6X0Ta\nf1BvJl2nzGsbYkAiPx65OBzRTT7iJyEVaVgn1E9/6NnfMuogjskoFgaMqxUZ\na+i9pXMoIra/We0YnF1M5hmBDZPP6Xz1GG4dVKz2Ze8VcU2lLPLKsgjnfwGS\nGQI4sx/EABdAAAC93EWU9u6CV0ZQnJxuNInqXjEkEodtJXDUUhxoTKSk68Ci\n64ZUOw6/VTgYUXYKHYpKy544tdhGyMSh7y0jB9HdmBlt9SXrHg+vZRblpGh3\nR2m8Let/M2CbmVMC/ICw4ucYNK3iWq6/d2ySf2VRP5IAPKZQIcTmMK/nESyH\n3YoMS42Ck+XewZmfKMktnpQ8CZzHtlotllwRy1o9r1lbYwcs3cL9a8fv/Hz0\nvzQ3GUObx3wDN6ZwdcagfoqdM50wA3YEbCh0rUoDuYiShL9AgPcwNTw/amof\ngG3O\r\n=KF5u\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201029.e864bc8.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201029.e864bc8","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201029.e864bc8"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201029.e864bc8_1603955341176_0.8233556940396918","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201030.881c37e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201030.881c37e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201030.881c37e","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"77ae595d9dc9328512badcff47168d664d48c40d","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201030.881c37e.tgz","fileCount":42,"integrity":"sha512-xlijWW2neL6pHemLIc907npnKaLmt2pSCYngrpMv7WJvsfV3SXUdkMHlY2XNtLV3JvNTJGkVWV6x7mFZoiXzQg==","signatures":[{"sig":"MEYCIQCzEpcEF4sAuaovWplTWtxJ9BoIf+knnQxXgfH+pEHnKAIhAJfCBr82GlofXCTFbT8+VHJ7QtgaYKBQ9+0DIFymN2S8","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":152558,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfm7wgCRA9TVsSAnZWagAA49QP/R1eHoldq8+aKpo+LjfT\nQhlALhRGzYQ+3n5iRMLdrHuC+Doi79vOCKOV1MPt8ysNvKO1M4+Dw+x2HBxD\nVrqXAUUMXxCNXPLtUQJ9Vcn0qbvJjZHOKHcB6NLVJU5lV21N3uMewbZTvcJC\nKbPcEgOqw+4nVNChbn5cG9LAcYRhTdLTKFNYTTzscIhvI8qAiG++KUrC1kMc\nht+PTuUcA1B6KXoPS9BYWlivn7dT/Ky/xxM7XWPpG+aa3fgMk1aq59hIwLUi\nx14IWxeNGg8uBoz60i0sac/f6oPMdhVtpVf+sjLFDEREpD0LfjftxLw+U47K\nIf9pEpex1hCbMCx7uLu/0AGm1y8U7M+q6O3DRSTDk08X6FiGIjgfItlf61Lt\nL2esIAsXZJrRIxInCVX9UJUwO0iIDgXU/fMIIu4F0srBwCI4D0d0xKNPQPEL\nIx8H+rWlZFOTppTivKM7teo0xmeofQDniyYINJnMt2XhBYkqafO9hol0SI0A\nGl2tx+jeSe0XtRMhuoSleEyuBeZ/fIcK9e5UiJNX1CK5kvKn8lp0AIdFuN2J\nHTdTT2SBbm1FE959R0DIynwkzJlmPZHIOV2gZx0Rwk0a2WJ8Iv82HARc7zsU\njwpuVXNuULRkX9BH2pZi5f4L0HgXR6ya/o2FF6laDs0BJWbp5Ha/+ewMOeEh\nF8Gq\r\n=aRN+\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201030.881c37e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201030.881c37e","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201030.881c37e"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201030.881c37e_1604041760235_0.17039272023330998","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201031.17d6e9e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201031.17d6e9e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201031.17d6e9e","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"efb47322c8664fb81cd1454b5f16ee15b123f2aa","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201031.17d6e9e.tgz","fileCount":42,"integrity":"sha512-byabpQMpt7vdk7/VUdwtjTTEHR6M8nU0NuHIjEf9wiJneYgLA96chBMD7FpUXwGTAtEKOljGL7BA7zhJxSXXeA==","signatures":[{"sig":"MEUCIDwZeCUFB8CV83LluWE//lB+CCj53GlzK0oPWkFWwCAyAiEAh9+pYk6kp/FmFVDXwfAOmKrwFjveSc8SqYIOrsULQhE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":152612,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfnQ1uCRA9TVsSAnZWagAAsl0QAJjsk2levm+oIe8oFh2+\nwd+QA7OBy2/ZkAuzy7y31v6G0RIsIdogrYS3w6I2x5H9FusM88JbdxEIFogQ\n94k2Nt1HNcaixGkKdjVpGM2o24pR2qNFAfgmpQwHKw72AN7ATJaQMY++xkO+\nMWfGzSRpGyq5XUEbXaoInuh5BUnDmrRcukwS/n8lHhbKkHsZWRk5N0L42oYw\nFycSqmNNtmVWKS9M6WLpdYN5OrHYExfITN6IoYDOjPchl7IE7dxq5/qJHGkW\nH7YMpbq/TDMSC0BcsavCzolUkuQHQc4EdPwPOYsW2GEP02N8fpddGUrwtiUP\no5Xxjt6okvWSFxAUVxrbir6e3/jF2ZGocMFwI88igph/scjru0YBPvwIgDWH\naun17OliN+++cQ4prYm8CB/ibzBy3/KxukdGdvLRntctpIjVwG61NemzAtfR\nQY35xbmaNg2J2GHdNErPEUkIAy6yUCrLw4emfnCe8iBFofAGobvLKMwlMahc\ngtXYM1GraVAfaTPvX+/rhEX+++BS2R6FVhoKGIlHgrCiBpF/P+toz1ymLCc9\nyAghvwZp4ylbIBhU1VmMVVhG/K2inTpeHuoNffdroEp0ZzGE0jSXcMRnQW/C\nt2dAuEPNW75HgMiLLlJ+SvXeTVyvbzZXME9KgY9YF0riNj79zU2ljwIAC7WB\nLQZ0\r\n=yJJ+\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201031.17d6e9e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201031.17d6e9e","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201031.17d6e9e"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201031.17d6e9e_1604128110070_0.5237888324488256","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201101.07f38db":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201101.07f38db","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201101.07f38db","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e89c302ef7db85417f5a0ec2464d9f738b53128e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201101.07f38db.tgz","fileCount":33,"integrity":"sha512-O5oRSWGK0u78KHXDCQWnC/RMwUVvdpTTDPuIbNLYoG+bqjSM4yITqVWaSlEk0QHQ9ZXDPhdFEkioPZy0akC4sQ==","signatures":[{"sig":"MEUCIQDCHpkt2otg4PUkBKXwvmZ+soNugiwBpabNzs+1V5yqjgIgeNF+G+X8UpmnK3nHnXHDkNNPThcc2j9TeNlnFom7R6A=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":122873,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfnl8RCRA9TVsSAnZWagAAsWsP+wfA3HKpSW3FulFwh770\nsIqiJUMFg2W0oVaTxIDNKgZOxntsgJ074gYvaIG1ejddBgD1R7pX3PxIkyUP\nq8t9iZDIU/CPuol871Ci4DIklqzNgIS4JoQY3kQPHhUdND08xurdQbWJ8KRU\nPcR04gdoPiGcWIufuf95ETt4sw8jZ0JfdhAK3taJ3HevCyLfYxPlo3RLdttl\nKriSg8Qye12PYyPs+JnNHHm0nz7tC4WjMuWdVBjenuly6coPMyiTDooqMOI0\nd9bewAkUKU8hKEC2usrB/leyMI1H+YV0SkROslQOCelv0W83yCNhUgkn5IcD\nKHyErWy7Ue/Q9Y4Sw9yfTd2IHYi4QyorgG7fP5A034iyJMSUnNl5A2dEs1PR\nSXpIDTxW7wg78+mNr4YFMr0lO61sCzSfOHrL+F4Ls8vEhL2F/SCBhPeHLYOZ\n51B3A0wE3EQ5OHzeST/5GGanFTZZJNmrydjudHNdwnZ5JLdcLuyiQ9Ql/Ebg\n8pDEQQHSgzuwvO4tQh3J6cKCTKmztnDRUr0iqsjy5i3wb/TxlywU+T5OsFfo\neVq1MnnJFyaZbjfGrW/ysBNDG1fFxijPpD3RfLI+69khAbglcEvNnU+yzo1l\nWzEd3SgxXU+FEbh/Vn7C82DiCl5m1sji5qjbCnX3fHYaf+G6MgfTT2o0Y4L/\nRJsr\r\n=3Mov\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201101.07f38db.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201101.07f38db","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201101.07f38db"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201101.07f38db_1604214545332_0.35723501098001886","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201103.f78e0cf":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201103.f78e0cf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201103.f78e0cf","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6873ebeb2a6903b574516f6b0b1e4093293a765c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201103.f78e0cf.tgz","fileCount":33,"integrity":"sha512-Qtew1HIjD1P7ZLDNtj9D7Vxb4xbQPhxC0xw0M+E+JmP+J86EIbnt57jixRBUU2r+iBb7yPiA3LWED+hAmwutqA==","signatures":[{"sig":"MEYCIQDkhNj+kQNFpeRIRwkxVUmLN4ewWQ71tqhtpTn5+XZN2gIhAJ5hRhpfPJmA75ukxw6GRNACxoDX2x5JVAQk0eO5xRzN","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123878,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfoQH4CRA9TVsSAnZWagAAykYP/2ubbLg7/vVyuhT1tmq+\nloDvoyZl/Bsw/enIOhqGSLuA58U7omoLjOGf2Jhy5mFOwp7Slx9UIdaFp+zG\nJ2XrGA7EA8nv45UqFLf6u7TcBJdCouZgvaQ+ppRLSHNwNJlyxLI8yJZPUKi8\nitWmzdQspx41wccrmwSqG0N5xNr37/kvRNFkLDfTITvgJrc/gNHAZonkhtWQ\nKyyUaWBoK5MnQqPeUhgOjOrE8nN2zemIkN4FFznMkF5T6BSgTMORfHYCxmp2\nnRSSLYIEyGgiFWnJpphUisFpTj0njuvH/GkmB3y/fjXf4hnRtnm23uVtmBMR\nhr4Pnhk4hwPP1lBP5f4UqCElR72VOxGRCsZD8fC37NYwVyKIh7vK06u/UtZx\n4Gdr79Be5hzzQUAxJwssUECgvwCW32LOtuFeMKJVSs958ov0CqK2PTYPXAMu\njZSWk5KnD4rEG5mMFcwvSkVptqwjQSBDkrAUESChNks0nQCo9JihqF8AqUm4\nFg4n98ORElGSWplBXLmiTgFkBiKTIV3SnVDGiWvwg7O864gTa1qQqHYLWoYh\nmcY4NiEive9v7Oq1pEcwuOwMamp6GWR261OptM0YzpdEN8YrB9kA1UaCVncD\n9tZOAn/uvMDPAnb3+ocGvZ21Ko3TPk6na1qOv9Oc4vZ/kyQZAiUAZGChEFfg\nfR1b\r\n=Viz7\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201103.f78e0cf.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201103.f78e0cf","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201103.f78e0cf"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201103.f78e0cf_1604387320144_0.641508108716242","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201104.673f22d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201104.673f22d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201104.673f22d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"91d783171f8d16fa278e068916370b94c9d47bad","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201104.673f22d.tgz","fileCount":33,"integrity":"sha512-/tPtG5Q2H78Biu6YhTwzaeuUGYy+iQaH8bHAUXnCQt5K3CI5g1HYO2m7A3p6SDyiltPYm65veZ5jkAcfoD4L5w==","signatures":[{"sig":"MEQCIEMeEu4KT3hq9kzRZtSiu4B727e+XuVMEt6VhefE6YOIAiB2gSeo7oKrMADs4fGdiBpD429PTICblTQxWYxjffYlsg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123318,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfolPBCRA9TVsSAnZWagAA3UcP/0eIPQKxH+B2PIISRf8u\nf9HDTbDmnPksHeOKnPSmQF4472NJeOAGg6yhut2aTQtWG9rd8mWrPFnJF3LG\nHdRD4joV/cwqLTr4Z9ShiFuSI29S/I87pPWcyFncTicfg5VwyN35KzVFNPzc\nbtnFxoGxvMpQ/36bixXAwNgt5in9FpMC/p29t7oLqZg7x9zkOFXHu+c/yL7X\nXvCn2VCKnDbMw0QPz8HO12Bxo5tR1T9r4chHSNiNcDKG7X0rB0fjYI5MGV8W\nZG+w1pHW4MGqAjzbkyXqTTkSkJdf02Tx2dHE8rcldQ/ALvQ67D/vKAcC+Pbb\nCJ0ggc8+SPYzD87roynHmuNG9/CLCVvbF2rb+G4tsPumR0L9bOu1dYE65vEJ\nl54HwlDuKrVlEaaTDqej1Hbt3qA+v0x0YJ9RLDa9/0F/IG3++ymLfn0bXRf4\n+i56hVs6DL4onH2epRI/tczyvgpSHhcbe9LC3s0N7k4+7WeAwPERQ9AttULL\n4Vycd086LvqAxPnwBQSOMBCKu29PoGZVSl1/TuPzWlLyRPM+EhWl6Q3ahIu0\n9+mmIi1Z59KRznsH4HOnFRoq2gdRc3u5pBLcMTyghmmFOXoF4sdZnMjqOurP\nl18czQxsr4HMOe1XwoPVz7UoUi1/UJQBQ6rkEp3AeEAIIU7OcqQaRWK1oo26\nbSxX\r\n=aFdy\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201104.673f22d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201104.673f22d","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201104.673f22d"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201104.673f22d_1604473793029_0.6663878956264333","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201105.0cc03df":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201105.0cc03df","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201105.0cc03df","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1b5c9044bb73449de88b809efc2818c6a4bc35a4","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201105.0cc03df.tgz","fileCount":33,"integrity":"sha512-kyM5HL19vihBcqCvVIW6fQJ5Mux6bzF+SM725afuaQoDhyBHZPpLfF2IH23FrFtqlZh7cXmTz26aSFQAkHGDLA==","signatures":[{"sig":"MEUCIFky2Gan+ZzpV8SAAAsNbQY1IrVKAfY54tcPhSAaQ2BgAiEA8TGz3xD+kxgLlHgvF87P5b9GnqAkVnQHr8D0VFPpLFE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124085,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfo0PjCRA9TVsSAnZWagAAc8EP/A9Nxlxa5OIDhwRw5VAK\n6ryDNQmnpgkwA2TGQ2WdTpGLqHKmvQGD9Yncpg3X13BB9ZFndduwVgpQnESr\njHnmarOj6VJ4ZCzgpfZ1TOlLj0Lq5tqjQ5U7my/B43HpW/CXwdp4ALlzqXDv\nM4sY0Sw67Qfp42gGfVvogQZiovgduL+tbPEKV/ynTfnsBQAbr5m9Je7EFrNi\n1rbT8UT/D7fBBqmQ4TKQU715CxoespdCv3uTyXAVsiOPkImbu82k/HfDviBl\ntnsZQ0CJd9I7H+snZ111d+ega7BbCyvXTSmBsNxkMvSeF8R/4RIO6tGE+4mf\ngpytMH5flwkCv4eTP4db2edBCbQOj8x0uybo+8jV2y9S3V+Ra8qC3FqCsRGa\niQW27cbTTMA04gtJ0miTh4Zl6lqa6Lv+hYrg7Vdct/r1T1QcUU4vnzy+bhLe\nTnE6RXtg9NyZrz06ZHKueX5/EW4zx4ylpL8Gzbg0f+v+W7IhS7f2NhDbE00Y\n+E8KEvwAHAVuApwLV/l8HC0Ps4J+eKnQ+tX/CgeThS7gcSW/9UDu1O6/L26N\n+nrVNqMEKJAEnXHQiihSrg2EB8JbA3vlApjGTPsS/TDhVO7F9GNdCVnK3U3v\nA2DiGCh9fuswBXP7TCsvkObu9XeI1n83579EBT2lDGTu+D1Lv2ZnvMPrAiWB\nnrfl\r\n=k5Gu\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201105.0cc03df.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201105.0cc03df","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201105.0cc03df"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201105.0cc03df_1604535266936_0.317048004644531","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201106.87dcae0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201106.87dcae0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201106.87dcae0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"cc7ab1d8dfc73a66d4a79477abd8b719f15c1012","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201106.87dcae0.tgz","fileCount":33,"integrity":"sha512-6uwjVpVqhxXmCtCy4Oz9xsQ076yAqfoJw1fC4k3P+5sbYh2TEselEporswh1KleUuGbFpiROf0pvbgt2D2ZRmQ==","signatures":[{"sig":"MEUCIGt/EpJf3r1wEPwgUpdnkDjOsBp35bJ/hr7VfkiTcoZxAiEA6IGtpvmWulrNVRJi26WhoDSMs0XZc+CtsVZ7BpdK78w=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124085,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfpPaiCRA9TVsSAnZWagAA0ZoP/0acTdr/q2RavcPBxOFk\nVMAgNUrMhQP0CmpgiLz4g5xcC5DqNxH7twkmDM+DjsqOwZq/kZvWfXZ2BB2H\nFI36/9P9TtXAdGq3JSMfW0SAbSy0tDZDSugOEJiuMNAL5xrqtCj4sOpmpzQc\ngl28ujLT1Fr7cQoty0B23FgzW5O5P/h5T4YTeDns6S5gmlABdAfXVRSJUgC9\nfahucokiuwH31N54GxTzFlt1/29t5/BciVnBXMrkp9VGdI7yRp3RqBhOkgMY\nw9qy+LK6EJbBv8CCZHYNwu7AkYRCfwQohT0x419udl0MuGvL9o79252pfeqc\nr6qYTN6+Xl+hqB+qeHW3lxJ9qkggMzPJe87YB1c01rArDNR75WX3Xg7qur6P\nobWTBwUMv7YqE/EJ4v3Vuu+uClkfrSbaBn8DoyctEh+Je/LEjt47x9zWivag\nDSrZ4/fBX5yN6I1qdxwvWzrDWUXwDMeHSE/Y3xYSbgV8/JaM42oN8FRaKBqh\nqFU6QichMXE9JN358Jis5b+U1d1pxframPPCnaDwcmh4JbPSSjxbExBbo8m9\nhTBlapENPrvCF+0LiBLmCQuIGwrj6VRwa3ZNvROFwudhSUiOLt9qmdwMnL9b\nqA8ODAH9hGAI+HIQP5TRxET5yc5bBWf+i4pbpGAXU6viR3383IIMV5glHS+z\nQPVF\r\n=xIDl\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201106.87dcae0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201106.87dcae0","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201106.87dcae0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201106.87dcae0_1604646561824_0.1865854918065386","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201107.23b4af1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201107.23b4af1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201107.23b4af1","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"ff0e1ba2caa827c2e792b16b642f2d7dd929abf4","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201107.23b4af1.tgz","fileCount":33,"integrity":"sha512-4iFRbEKkqzn4IjNE731tm8DE3tJ35lZmjQ63eJGa7xh6CqObPgHDmcrhXfzqUPkxshGli1f87AxZK0sG/WNQ5w==","signatures":[{"sig":"MEUCIQDq06RSXlECHe6dWm1nrw46g/MuQOUjTSw+25+4oLo3iAIgSNDdKVfbsNOqFTaaafwFeLSUATAVVFfQ0dBaTYF91BM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124085,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfpkfxCRA9TVsSAnZWagAAQNIP/RDk0uTrYB3sK2vqomXm\nYkPOgDiBRxbZa3KniBxwgoTNbF50oYdIgD2qGYj7U4ZyR0mwzl+bMXp/yoOy\nNpN4VyV+cclVB7p00LfbpGokxCrGe0WA/oYuyWfQ+UY5IETOWSuIryMqTZBi\nWFufRw2X4u9pqC4dF0XXfzIgMFvQbfk1VaANOylQA4tUryCjyjcxKO3I0zw6\nUyU6nQF3E1ul1o2NKnb9jiW40HQBIRgqyuVlaxjuJeL8FMFF7xcMx7jSmt6j\npoGOfDCT06YPG1cy6hooXIxcfgRvCUPnPAcl8nxMSHwJyQ1DoMOeYuWEdX4K\neuynUP486cxU4klNhhMYli85eUgHv9TwMZ4O0+ksbeMVAzwGf0oQiw8b5MuK\ndnuiSYLxYHcU/FITJn7ZahV5Jy8x2CSbpmaS40qb5kxOclQS46d8Fslejvwe\n81D2Dhu/cjleRCsF7/Mwfl4PuJLDpN5hL6uyL2UA6LlKntSckV8/O5cvyr82\nVrsuqhbldlIr0zl6P2kIYEWSwL21Bdcd3E1L9nmUKV4i+p6pqhFYQ4bBf5AB\nuZ1xFbqC/Hesz9YsvhyuiEIX+gaU92grT5B1VmJ5Ih1lEAHl83tMcf4/MVnD\nsNBAIYmyIiQPDVb+A4/U85lHNLYH8QjDxkGMHhT/NuPtHPJswiChdfUU+7K4\nxgWr\r\n=Enoy\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201107.23b4af1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201107.23b4af1","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201107.23b4af1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201107.23b4af1_1604732912758_0.6467866664934316","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201111.10062c1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201111.10062c1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201111.10062c1","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e2f2127188a84597440d5f5a2b343208bfff3ba6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201111.10062c1.tgz","fileCount":33,"integrity":"sha512-jeHXKIluUqGeeuAkzfPH7fq5/8mVDndvbVuZ8FtCg/0aFe13TY62jxZj/4xE6EoYIR1/7awxSc/FAAtTgZ1y8Q==","signatures":[{"sig":"MEYCIQClXHkHSlkR7zoTHnMtMtAPT6a1rsha6u/DFdG8cLGAfgIhALC5qSgS/c5go29E8MiwCiW7y6cyjCdfYtaVAY6C+0jO","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":124085,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfq440CRA9TVsSAnZWagAAd1gP/Al2v1K+VSHJX2IbQsCw\nACUQVz77UTSIxT9GiwShp+kXl7z3+xkr7J8D5lRqdxqCnjblxLLl0fYOmZlA\nSmBbuxB33KQVwSnfokehqTymao+4FgVWVwY1IOZquZtaM9Y6r78HTMYBp4pw\nSMTYu15OnyQyiiD8E7DjQfk1hv8ZDsfY29NyGb0H+xbeg91LPRYJ3L/smhkU\nU8fGYZcCPuLrFpF+7RGfIo5zaUw2yVZzBnQz/+RvHsQA8KCTuvq+fhnShelH\n7cGdmK4V/5QpM3O076AQzS6HtuPGfj8Idjw75wfJ/BdlmX/nu+zMQB2O0Z0G\nMnVHTR9IHyoaqKrelK51dUn5dE+BATWRvqc1yppzvULpQr12GHf9ZNr7p1lq\nZwH13g6vVmk1SZIK3VdniDyiN+LbPNYcUAzR0gtsNymu4naC1fmXFKEKvFV4\n2D1KqyKKJhxYJVN35N/H+58UVP+QjVWKKok9lQEV4KHAbCTKGoMQiACq7Aa0\nKdpnkniwsdvCRSqewW1izZZtUKmNW1bHN7XnsIVTRc1n8I/WhA7a9bk0kNlm\nGcWeIgZttUq8vil/AYbZwoJPSpaUeaQ+RcnQUZnCgwQ2uvpgTTEiKmWEKvLB\n7RnuLTfgyqH4WdxtES0/VGj7LjaUOwI1BvOz8KZ9hqYqVhMvaUbk8KwDI7eJ\nxZCU\r\n=tzh1\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201111.10062c1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"12.19.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201111.10062c1","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201111.10062c1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201111.10062c1_1605078580436_0.46720806116350766","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201113.e85b53d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201113.e85b53d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201113.e85b53d","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"565f3783473f10321a4d12cab7d071c9f85e26f5","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201113.e85b53d.tgz","fileCount":33,"integrity":"sha512-B1PIPulFUjQI8yjnBzNRQ+OjTy9enuUMlSf+jMDSXpjhXkgkEDAVREiKKcPY7HYRTJHjPuAlZTnlPBRplJ2SaQ==","signatures":[{"sig":"MEUCIQCtjLn907kjSBx6vIMVxPi1LOPamV5a2hjfleOgVtYiwAIgUkfpzlZ2YO8fkfy7w6oLaJvB0kwKMsxKujV8/8qmwFQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126857,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfrjE3CRA9TVsSAnZWagAAaMEP/1kUU/tlzxzmhoL75i8p\nyQxSIh8k2ZCBUCVhofrcvp6CmgUGNYpOGHv56pDWV1LfB92BatMxdgHz7qT0\nF2vKc4rkf+YsalAWe7Uyf4JhRetpQGjeyymQ9Bju29cdCmefkDI5o3yIprFQ\nBD/WuQZ9NA7wNU+SUArC8liYa1SKW4DoBsq3U4ZgTWw3H0FCIEdq8yZBpVM6\n3jKR8rSTMgHuGQ1sEJcTV0BCvOu4upy5LiTUwkbjKMt5GhFnu3601F8/g2pE\nwfFibAcMDbRtYj/EI4QgYMH1vdU0lnbOYLvFxEVvGz0wNtpOfYlCLGXgCvxo\nej86s30JpP0QYNWSvEDnaUyBBNtgkIRwQXR6TBN+gqBC5gPo000A403lXkrK\n2+/44lYSbnlZFBWTc2fRcqHThWnuVmBbsQp74MorUhcj37cVJyx5A3ZEsxci\nv4FvfIRGIMRCOKKZYSpJHBmfN8hNt2LdIdu+bFQGdf2wQpO3146CumYpoz4r\nUGQfRp87YW1kNPIIwz1EdFuzXE6tiA67AA5txDLNMc/rrGktAGuqLvGFBDZa\npk1w/FEI4tcJyFeXslzX1WhVRiZkrvRgV0Bb1UAGm0vQHP5gRugreZXqatPB\n4rKtbaWJRoBSMCqt92mBSYMTZSOQgG2bTpYlf6sD2MjfpjDMNoOLlPlAdJsz\ndYi4\r\n=Z9sK\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201113.e85b53d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201113.e85b53d","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201113.e85b53d"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201113.e85b53d_1605251382902_0.15523730536999358","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201114.d501bf0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201114.d501bf0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201114.d501bf0","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"780957c9afa993e2f7e8db58d92fe1bb2fd9c2ea","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201114.d501bf0.tgz","fileCount":33,"integrity":"sha512-9I4ZYlaFcC5ebQRMfDgH03QCvDzB2OvsV373l2gKQBTaJWKs46116xugdRVRL13YeMBssvcdN6J1qbBaHkvStw==","signatures":[{"sig":"MEYCIQCeNGRXMAzrN+vuhW+cdRiV+lMA2uUmhuyxtcfSyrG8SAIhAN77aWrTBYQPqMgu0SEegrTjkfnWQfHXeP0ElpdWxNOz","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126857,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfr4KfCRA9TVsSAnZWagAACBUP/27K0qFxOtKi96OU1VFN\nYgzFtwXtK28mv8zaZUffVFJ0rKhaajfivCnk1gGCRfSb7kgLzAwX72QdQRnF\nxpACMn/oZJ/zxED17B7kVi743Qbjl/mnbZVyrY4pqJwdjAFVLGXsdh+nQrE+\nUQQ+rmbRhFyXlQi8a5+sOwTSGLbeEh1lQLJ3JirmMCmpk5DzymS3Es53zGdM\n79rVX2OrU5oKf7valqzGrgN7cJhs5htgn8fYnNubiHnKHAoyPF0DYcpaoOfb\nUb9C2+32CxK13f2aYFkB7bTGAr/06XSYaw+0M4yF9wLk731EPyqglhnF6LbQ\nhp9QsE9enpetgMTd1G4Jl+/bdy3vt3F7WiwSI/8IQ3CmT9Fdnz5CDdpzvZIX\n3cioJyTyjdnOyBMi/hZ/rKvi839hxTdLK6UAmZEIhIjhW6LZhSouMMvagn1E\nr/Baq5aE2E+B5jIwVUatoqXTqANoTck7/mmsMeHoT21InQWXPw9N/Ez2I5Rm\n/v0JP3PweEcQrzXINoONT7bl9/TdgsSNTdKXDcE9mLkxm5ROhMLZRPez56Xs\nEKKasTf42u+3Kp6LsmuzNkW0CpQ3lNZdorVN9HgMmMoaeXVTrZYuw0nPI5rH\nElL8QvT8MtTuWMK2ItAAJrQkT5Aiy64Ew4f6ZtQ04lT9YQs48op7liRuAmt2\nQ8Vd\r\n=59w0\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201114.d501bf0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201114.d501bf0","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201114.d501bf0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201114.d501bf0_1605337758738_0.9765482662399085","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.9b18c73":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201203.9b18c73","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201203.9b18c73","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"bca558a2f61345306821f26d788ba84be49ee051","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201203.9b18c73.tgz","fileCount":33,"integrity":"sha512-3yQv3RPT7EWjUqR0bOL5ZCtoeAbosK0DpJJrOQb7ktFGZBdexrjrhCsDUOVuGuTGAy9P+CjNA3x09FUZQBj9Bw==","signatures":[{"sig":"MEUCIES9TLIu1Dt4I7YPwx3Ljd8wWzmdMzNd7j21oelX6Oq6AiEAunPJeE34rIz37rfkGMkV4yEl4X0t29215GknbtndX7o=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128810,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyI8cCRA9TVsSAnZWagAA99UP/Aoib4UWYP1YrnSRygcN\nUi36rPvHGNIOPiJiUeFEccOK1Whlqu29sGhSvKvtxf7b1mEOQaYSGJNu6BTq\nmuZLwwpnQI4spGQgTQd1oE4Nsizr3QrbvAAjeS7p5li6O2FtvSeXve2yHJNh\n2JTeOpLONAMBMx+c0sj9rXqWhfV9zOJGSRM/nxysrMZ4HqBzkzWkY/bMVlcU\nycpkBquGwJIM8V4/GnFqC2hDcUyMkQwyellIUQCS73MfVn+LaPMToEKXVkqs\nxS+00LvrN2MygkRCFrsZmWFpWTi3GIhShtq9NPC8RXzWt7h+jRX/covycSEt\n6nXRHDBQKdfZscNUP4E48+gIenkDg3xHW+tbTFmbbmY++rT2UMRXN0PBORu7\n+Z2Z2k+fdI3D85AeXUiea7IvFK+HN3ZMJ4HUiKu0v/SwKG3ZMOg4NAn51umO\nb+BrKmz3wj0VwtcIRmi9Ot7UPI2yxhxuxMUQKML2gUfTH9Nh2U8LEv2oextq\nnF8TuKigO6vssIISj2gaYjTCvxxwyNRdjdX0XoJtHReUgJtRNzQ2fuuafLHb\nNjPhZRkYBgFQrfNgevMEJI3R8TBSn7e08KOqsH55mSrX6+1Nl4diQdLYRsIE\ncRqLw8ZXxvOu6sNi3hgft+LoLZj7I10nkdaUQ3wnt1ngyGyxBosqv+NMeo7y\na4V/\r\n=TLPv\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201203.9b18c73.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201203.9b18c73","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201203.9b18c73"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201203.9b18c73_1606979355602_0.6578377199015164","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.69f8eac":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201203.69f8eac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201203.69f8eac","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"17895d29eade911850e5985f6b1a93c45c2ac5e6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201203.69f8eac.tgz","fileCount":33,"integrity":"sha512-XWGIYrcoisSIRby67qc8zonibRbIBmqJQz0zD8OqZI1vWUEdyjpPYXPkZcvhTYZqZNKXNsIFSugNkB2bZaeqUg==","signatures":[{"sig":"MEQCIFgQ4GAEQk8HcJgXK5lZ/e6905Zccu9Jpnfbcd1x9puOAiARo/AlM05PNTZZXYYYMNDCGnLQ1oRw11uODf/fyqipdA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128907,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyS0dCRA9TVsSAnZWagAAE00P/2AQpZlbr0XLXGvqw3AJ\n72EN1rC5v+taTCrr+oOIIbOGVqNHwe5CuR8aGB4wDIx8RmOE8lBT0tNTZWHa\nTZvEinRoVPq3fAPs/Xe+mItx4Y0bzdxcyj7TY11/6mE91dX0sxq4bvTMCaew\nnJKo5pE/YbA44nUKkmM0JEhVwYaXPdLd6/P7vEjX0CsQjcS92CioXIANFsFm\nag4vDQ/aT6aqauPafejFic8Ii6Zrvg8crsF3dEcEY9t6rZZsT9tnD0yWXZis\ng6axMZD3T2nl2ObxZPuGdj1Fpc5m9+yUp1appVdQOd8HLxGnEykogd0NiEWF\ndEasqgAtg6QfourXL0qcu427DMWpJGEfFNJmwdFSgjHH2fhEWLpZ+HMpl7gJ\n0WKi6OPPAxK7tY927OReO4hBxfxJig2UKLZY86o1lIohyUsFWsmOqjxRE29c\njo9McAS1Zkhg6tkTccbVWGKVn16HHd9N1j16bRm1de6JeV/7E5V5zMtb3iLa\nZx9/l7bdQb4C1WyvvfiYZclqBuRYIJGz3H4Mi8e6xHQ6AEeuywN0CTuSJNlo\nk8KU3c0NT5bCRFtXSyZW7Npw4N502XC5P1QRKxzJ4RJ3nE26heNcNrtAl0Or\nIoC3Fidofn/gWCCTll9v2Zpyo++C8KQaU+mE8of+5NjhzRXCIAQBDpXZwPlJ\nL0g4\r\n=17dO\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201203.69f8eac.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201203.69f8eac","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201203.69f8eac"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201203.69f8eac_1607019805003_0.5850761346297126","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201203.cd29e25":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201203.cd29e25","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201203.cd29e25","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"e2b591315feef9979fa1e6cdb0729a8567c6ee57","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201203.cd29e25.tgz","fileCount":33,"integrity":"sha512-c8MA/G7n6sMhRofPTLPdokeCYVMYftKjJs2j5lpmtWRdYuGB8Vw7Rs6KGBZB6ZEMYFg57XFWDiLOlJ7byjjL6w==","signatures":[{"sig":"MEUCIQCSjdZ2i7mjG0Rz8ipZ53+9PSeUqNi67LnioA3v77sjMgIgR4Oglf+nMWr0HMjBvfu0LjHv0nGkPtp97du43mJZSww=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128603,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyXP7CRA9TVsSAnZWagAAlTsP/RYjtVq+KDp7MRBM/hHh\nkvsg+BU9rU/2vTKHRJLJCRsiGZeG5+t3qzJ4NG+OgzdVH/CfmmAIq7reyKtc\nYQBRMX0Zoovdm38fesxgmNUW7JLZ3M8laYwqDRKTg02gTfNv2dUPAA0w3pO+\nl3tKTvVI3OY5bR9J1zBgzIykMuepFpv0zKzuRXIJTNx3tzbktwpbU01FD4J/\nmZIXzYPe8DtBA2nviTSyM8NqVMJmsqDdA0op8iQq7ZdBRRYp9Z68bcgSjDui\nKqfPZ1Rhfw4rjFm1+VoZKlyr1FICx2t1vX84cT8K8QmvS6052km8C9Bvw4Nn\nl+qtwOHLWRO5AcU4EnSFALcIDnsh+63EjczqZGGHsW+0Hu4OGglpwDUo0J2q\nMMIfvpcuzeZSTO4DFCNN4tyG16Fuzn0JXHUY4x9Gp3g1bhwG2mVskTeVs7gd\njq3RrPClN5v5xkr3wCqTcnMXOdLZPWCbM9ny3mKAebe5ZqFSOEu/eVI34mYv\n88wGGlr7OotvbaqpSaahcR3gunYu06UBknDw9b7yWJzIdMXxVr30B+0B4Oei\n4YHbPJVmq0QCNzccS4d82F4WzMAJObTEvsU6DPytLlu8H70bcvrw0tznuw1P\n1qdp0v65gyyA8hoWrxzIHMuI2/85YPbh3j2/3x3c6YHQvDBd5EvTzdOqGjnO\nV01N\r\n=m666\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201203.cd29e25.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201203.cd29e25","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201203.cd29e25"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201203.cd29e25_1607037946490_0.43474343067222443","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201204.cd29e25":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201204.cd29e25","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201204.cd29e25","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"1802d2bce3d9859f8a0731ae343b65963db566a6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201204.cd29e25.tgz","fileCount":33,"integrity":"sha512-bP9iMahDPZ8hI9anMGXtH5un1jptI6k+ZgWe5hOx1mNQUB51o0WKhYtyB07yCjT2G5J84MZ6HCSqU11o6mXsuA==","signatures":[{"sig":"MEUCIEUPDhQKifdA5wfqmBjGxTK1YRs3two+I/Kjivdqb3HnAiEAxHZFAL9byzopKetcdMxIc7kIRDH90XaGoEWHLmFwWBQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128603,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfyeCyCRA9TVsSAnZWagAAs3UP/2IbrrXMgkeHmsnhvzIO\nWUXg0VZ2ylB/6o1P18MjhNEGNDcATLtnwVN0q0a9B2odkaDKQ50TsrLQ+O3T\nhNo5L3Npl/uVyBIwk7uGI4w04AR0hAEGcreBiVcg0j3bqepAC9JN4wRTPGfE\nj3zVnq2hcaRRZx88+q2b0Ro8WuFwrPllgSzUH6s0iEnJ7MVsoVtw1MA96Kvu\nTXKY4nom0HdKlmS/0G8VL97Y5cSeTXx+7XjKj+46bhasR+j9ydCMGR+9uNvu\nq27Ku6X1XeZ1LePx3Ccl07J1r57OUk0pOTw/jW4C6FaiDk1ugynrhMdPY8Xi\nXNLuTVCoPaMpW4KTaRQMm/is+xQNOHsicpkO5jzsaQAb3LjKZBd2a0Z/vWkx\nV9+SAuKOXlf7TN7hCtNw2UbCnMvh8qifSIRl7QMSvl1iWUt6xxY9pofKSI5n\nxMhW98qC9O+XdlC6zOEsgMoUWh4GcZbJEUHnCi95da5O1nq08csvS6U/iXtQ\nj7bPNwoRaIhoU6q8HPYJbIQBPhzwcsQspyCIZ6XqmOkSGN9UlvstRd9CBJG5\nfQk48poiak3jkCQhWW9L0ez8NoHtGjhI5Z/H16XY+7iCqEHjLndHBp8lrHPv\n+iRBwTBqYeMnpW4mNKeIzdV6sPn+4/dWPpJPwNq99s2NEJvyY/hG+ubLe7a1\nSUCW\r\n=OHYT\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201204.cd29e25.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201204.cd29e25","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201204.cd29e25"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201204.cd29e25_1607065777746_0.6830682572406808","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201208.165a6d6":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201208.165a6d6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201208.165a6d6","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"b1270523edde74c7f95bced0508b4a667bed111d","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201208.165a6d6.tgz","fileCount":33,"integrity":"sha512-vcgnz3a7uN/bdDyVPT832Qy0O6epTFtAi+Lu5TrDKhDkOMEIBYjCaNjmu0KTaFKi6GDTAYaxSkLtCbOYDPCGNA==","signatures":[{"sig":"MEUCIQDa5gfDyc/+CHVUNP7kU5bTqmK1jyRU3W34pk3VfHl7UQIgFoZruhzH1tVL/OuFY0pF2hIIIEPjPmpSW+HTDZkRFCo=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128603,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJfz9MZCRA9TVsSAnZWagAAWu8QAIdZDvwola2b3pa0owMb\nlG0MAJRSw7MIxqsaSYIrAryHG0Ot0TPpLIWsbLTltpEDzKpGRCyHQMd1TrGq\nqD1Dhv2oeDZNs5VbuZ7j7bex7TmjVwWG2ErArc1GHJNK9mfQKFaxauBbTmV+\npiiIqwj4Ue+utEmqCRuG8cCT1bAgmZuH8CgEHa0ev/EJt4bGnqirz3lVy0+j\nX9sxN1j51G7nH3G0ksU4gDPisPh3PfIyBoOaCsOt/OzjgNsn8ERxgS46MHFf\nm83kO9mtZSUbgxKTC7n2v+9SD6qZu8cagQjiBaAVkkNFYRRCJd+gbVdr19Rt\nQ5Zw4umu+wL1wNFLDRHCMK/01m8CmfUai30ok5eC2yIIre0ecTxWb7WvRVsH\n/XeUP0J/CsBwJoT1qa8jAH2ananeFAv/fQZ6dQtYP0JdRxOg6VIPUaVp+Y6U\nTTvA0eGJJlwmO9JZW/LJN49SPPRaRWji9mVz1sDAAPjTm4/NVG7hEy4OsP0S\nzKvPkSrxyX56HkYQSwt+JKMI7k72D+1+NQH/w80/y4oBcEoE/4qqKXYmTI6K\n2XD6YEn99joU03tqEa1UuDfm/+DiZiPTdZ5HDSBQ2Hieotfgk6Lg0AFHZ2Ha\nY7OM5DLkWPzGaK3f+fxbvFM5Qcz8rOVu/ZbKYKrbM7lO69BRgbUKNF6PKCCd\nkaJd\r\n=TfI5\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201208.165a6d6.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201208.165a6d6","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201208.165a6d6"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201208.165a6d6_1607455513240_0.41247450746873526","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201209.165a6d6":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201209.165a6d6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201209.165a6d6","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"6a1d42143fe6547c34d7387e2a379893ae879a5a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201209.165a6d6.tgz","fileCount":33,"integrity":"sha512-bbaT2dJWuBVXW2awLStw4q4mt/pQnnWjtJYDwx0CrDv0IRyclHPtK5HSDdHLPpsHe81KQ2ST8sDlPhlBc1IT+A==","signatures":[{"sig":"MEQCIE8jBBP528U1VEPM1DUDO5r9xlEFOiVkbRoekB7SctKKAiBehIogZUQ+VKe+r200WFA1LNgbJ3ahn1Jzx80gBbrJMw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128603,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0HgDCRA9TVsSAnZWagAA/DcP/0dMFlrwTWk/Gb3d8Gfz\ncd+BrIjFbCfzarO0GM5EgEyqNC22t4m18WW7ONqB9vbhn8RdEhdxWFONSox1\nXjHi1pIlh9gX8XUwZ9pfZV9UZdfzjKebVNqApLyXjZc7RSZxivvWt+8av//H\ntapKRIDMdvoBmh3Vjx6QYh6wiFwcOM2GtEiQtC5+xswMVPU13iwCItWB3e9W\nd3Ph9J+JGs31lH1IYO/EuSahd5LT2otL/ZVI11MP11Z751GBbQ+LBglm1XbC\nVmVl/e6pbBPslWA+gGr6+7wt+It166EU4DGsTqUOPEYqDZUB4o7C/VwpmTwz\ngNuPBby/654TF4WyuWLxQSh+mS8UXmtDhEHapJ5Yvw79B1cBwWDmOPK4TUu9\nRHgc7h0+tVkObVdrHTyFDrDl6+YFyf87JnD9KDLkVuLq5d2iVu8+rWwnSy9R\naNhpEYrhMRR0DoeqGanI0pizg46JE2vAiTtcWwMcbt2kptQpam5As28evqzl\nYSc9Ma3OIZhsNfl+ieP4sEmqQB+9SIbWoHKeEqkUnmGh99C+TX04PXg31wkd\nyfP7iy8N0uHk8HU3VC6W1/LorEhA/shoY3DIbyLmRjCp2jHwEBhRjJw3tN7W\noCsZPs9+gGYKvDf/9CMWnBX79hASRcisyFq7hNjxnOc0GhkShP4+gRusCoul\nCfd0\r\n=uySg\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201209.165a6d6.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201209.165a6d6","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201209.165a6d6"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201209.165a6d6_1607497731324_0.6370711893291088","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201209.54c74f9":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201209.54c74f9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201209.54c74f9","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"71245fff43092992993904b4ee34f158dddd0fd7","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201209.54c74f9.tgz","fileCount":33,"integrity":"sha512-WYhgHEIJUEctIjegRxjmJP3JP+elzzQ5OsNSwZlEO0BtHbuSgWrt8MB55x85YwN2kikxO3zKs6Bx5kkZm2uOPw==","signatures":[{"sig":"MEUCIDVdS+NYZ2lVJj1uoSARdAUbufoPwP7y5Q4odD6Kt5VjAiEAt1mi6wGOOgzbGKUQzWbD97xXDwWz0bMdqbPDsnqbSR8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0TuiCRA9TVsSAnZWagAAM/oP/RjVlBDru92Hc1J/9se7\nMTkHnJxNjmPNIwl9XxSl46pW4SbcCn2KX/RTM+kWlhMEYDiAzuLI7dh8nCmo\nWucdkLrEFpSYN7auaZGZjZfMrj+MG89oUuuv9TrZScqMZyW+/LWU+l3ASkwP\nPIlxqt4gGKUccJING4q7SjT4ST9QqzVMI9jh0f6JFU9zKqdc5+zKO94lw+Yj\nTGeFrVxtlfBrPiPEK+0FaHMtPOTU4UYnPNuGpLfG6tBuEcAhGg9w40uujISj\nsff5KhQGU5eD7ZCV55JVxvoKuZQLX1ndK/2yblaCKFEEdyW/9Yg+Yxt0tBsI\ncWE+FCavNj89T4kxKvvw+CdyFGGN+mkFtDBFdS6h/lrO/lkzptt2D46XAinE\naSXbNgBMQxIVSkbMvR/HcnPCSM0eb5ZPYUUx936YunsjRzsF2U9tuICZY590\nysGTUUZTaY2nwjc/RKEXFn1k0mL8RAbEPLsDZqkZ29J8hQIZw8w5H7I7kN/H\nlg6OdnTgLYvyOrtqIVCpSrgK8scwqeA74XI5+7DYDwjO42snOuU+YpYFX+oB\njmDjKJLTqU9PtG+3yj0wBLbxFb5R44QLESGrN9OlhU0mzhs2TRP4m1Nd1mB+\nr4Cch8TAn6ax7Go4OclFVRL0LVTz7Xuveh9jAw4NuZA4mQNzom7wYHxcme7L\ntvOD\r\n=oE55\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201209.54c74f9.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201209.54c74f9","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201209.54c74f9"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201209.54c74f9_1607547809617_0.40530926788050303","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201210.9534f1b":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201210.9534f1b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201210.9534f1b","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"c5434cd1526f7f1ac98c6536c2c323e32eeccc1c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201210.9534f1b.tgz","fileCount":33,"integrity":"sha512-S+SrOOSAOZiwRR68WFP4hDztCMS7hNIlIp2OR1866G4v73DWzbrKRFPUDxF3Sz+/cT3RaItrAAmpow2CP+3QKw==","signatures":[{"sig":"MEUCIQDgKVc6nbBhFAXOxFAYOt21+r3tTdIJuJHC9tU699p6sgIgTZ/EsQrSyfuAZb3pGdblFk8W5StA0d/WIOy8Ph5sa9E=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf0cm+CRA9TVsSAnZWagAApOoP/1VFZCY3ILVIBhsySLtA\ndIC2d+iMJTFqgTJUVEwmUGT579mwS2/5VBJF+D/KONUX0Bw6p+3RBi4Pb0P/\nDnFqxzG7HuVpEcZ3oxPZoOHPT7GAsn8rrUp8Ii84TyIW+zIaRt7924yewAzC\n4Nng4wZBrVZx4YmEJkSevoxHIzBzENd5j2j0dBz2M0Y7XYtud06gVb8FVQcH\njJZ5TwndADFwBGj7KrJxW4oVKcaz/sxoW4csfOPxBCf2cDqU6R4M4dq+ypqL\n/lZS0eMZZOVPEOJIK99yzN+traZH3KmjlAQnFUvBP4eG6NITyF3yMqvS6WJt\nvVYoKPxgLDQqCTIfdNRdnRwvhdjxICHjWZUPvZGecIZpSgIt3XXfPbfqpuYv\ny+h51bNhrKfQFEm/9/OQVppO3LOQIKy8Jz1t50xKGumBwZBL7u+iVlMaYu3Y\n4/caL3IxrqDDjq8LAIa2g8ORGhRkEorJV1aHxbJoeVDLbq+7aWzdh0lL6Ua9\nyGS2PF4mu0b7V9JQbctuPJ6RbzyHiyXQLePS31BQ0EgEatdKkKv6oURxrDC8\nXL16wEfhE9FzSJXkdJQ+nhZ/RXuqMbhKMLYkTcVa5bg70XMqRbDkFJpsjZrV\n/1YmNjDFmfXWKw0gRRVpo+GE1+KmheeZp9Oh9qkGmjQzh3haDAEYq6Q/QAfp\nXhom\r\n=6LOq\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201210.9534f1b.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot (.blu) file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201210.9534f1b","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201210.9534f1b"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201210.9534f1b_1607584189903_0.8075661346540204","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201211.b852646":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201211.b852646","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201211.b852646","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"78d8084f39bbb6d4bee90194f07f0d810e6ae248","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201211.b852646.tgz","fileCount":33,"integrity":"sha512-azjmdYOqBFMocFjVzqRdGaSMrkzrnvSg0bP4KonWl5xCyKprzDy6e2HLYmbnu2NvR23dMdEt3uCbPE+uysgukQ==","signatures":[{"sig":"MEUCIQDp+n4GbEIXQRDOj1iXKcQcpgcUz3jblZiVyINHte65vQIgMTOI+tGKCbZ39lpH4A0fLZz1uoTNqpf0AigYw+uvO9k=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf07XTCRA9TVsSAnZWagAASekP/jZ74vhTzGUlD6c0PWny\nF/H0DzCriDFVYbDQsso23WW4S8t9PO3GJe5Gc2T3Z5pRDzJjCtSMgxm46ENg\nytX6dDWAdJKsFjet4Lvg7KhReHlPTDY4T0UxScqfC1D6YWC6WMks4zq6Qc38\nZqO4Okna1VYo6cNdiKvY7Pgb468OWw6NmMabduaF1PmpO0hIYmm/5uOLyN5f\nXhcTVPXT0/fk3PORQh5w11uZLC+C0dSdUR4ADGOGkhrnv9Q/aAGuDVenQX4m\nVsdeWpHxeBaC6vdFXefHDE9PXrglQ8+16CSjmn0IpudIywj0cF9D3TzLdz28\nvww9sGIXEL+gfiN2j6EEQYR7NS7Zcvagqatk/FXRQ2Yc0otPas8h4v50++tQ\ntYo/8XsDE8DS9h3OVPAPiS6Bvs5XWpkUVaewG+0IXX70hze1Nxk2Za3Gi6G2\nxjiB3XnNcGm2ttZHKa91gXwDRuoyHYV5cq6sTzFMzGCuLeysQbeCs6RXNi1L\nVDDFQPn5sGLjFU2Q3eCHKgLXDbRcWKMjvo9or2EM237NwEcTxIhMXIivU56/\np4U1mObARUdpO2ylzXwUmIvw8OQ8DEqFHN4YxieVL3f0TlYKxMYKBXoCQ9QZ\nEtH4qmTWsHAtgR7y3aQJofUP/v+lCnim58qsxljzvsk8MZy2ZXNvmtacTsC8\nXrr9\r\n=IWRa\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201211.b852646.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201211.b852646","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201211.b852646"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201211.b852646_1607710162592_0.5459250030154468","host":"s3://npm-registry-packages"}},"4.11.0-beta.20201212.7d972c3":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20201212.7d972c3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20201212.7d972c3","maintainers":[{"name":"microsoft","email":"npmjs@microsoft.com"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"5d05ceb324da037ea3304a21a96ca8b0eea0bf3a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20201212.7d972c3.tgz","fileCount":33,"integrity":"sha512-XcskUqUuVSzwjqAmCnRBQ66SR2JvBqE776qxc40SOPFx/RiGAwJmqHtTBpxMj49S+CNZ4jzmQN7S8C24Ij4uaA==","signatures":[{"sig":"MEUCIQCRZV3EcPVNXcJVQOC3kXvjofkVlSJ2+6KiKWa0Dwp/fwIgLJ8S5DVRI+VCMv//gRnWCRS/fEoT6dFazWgLM4QouF4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJf1GyiCRA9TVsSAnZWagAAYDsP/1cCCs2w1EHimlhDMQgD\nmNUujupDORDRGFjgKe8X/aKJnzXMnwGIjKj0kCodVsri1+Nnolxj1sv8ko0a\nmXNV6/DaT5QEBQ2sa0O8u8j7XsK3Hme3/12WWMom0oo2SxetmX3YFdDK+xV9\nxncAPw6mNncUpS5KmatVyvHFCTs7DY6ilf+m75z1douilS4zN99vZtvucr86\nDsuTK8vJxxfAKD/UR0TlT5kdcW4TeqpRZpx8eMVgNHSRZ7mKN4rGK2inoxSK\nzj9PtX33MwrEL58IHW+7LlqsUb1H+X/NOYZ9LIWWH+gqijbE/h8kzXsZSHhy\nroRZsNgMtOjImsHmebUbCndUPihiTKHgfZB3lJ86PZ+KNF793Qri1DHsNJ8d\n7irNnqCah0Jne104HZCrM4AUwkVFiB4cmyHfhls4tFYNKLC1W0haOtxK0+GZ\nqOLkg1eQRtHLWdHScgumHOzLL6F+QauEI32kB/Ai49aRy1LXjboDHst8tlBN\npzqCN0J0YiDBgM4pHrroV0zvp1tz6wtE9rZ08lC1z8Zvz6nHB2TPR7eDAOnR\n8edTtfiTklMo2tXyJ/9O2tUsv0jgWFjEBeExZQzU7T7AEVpA5brpO/EYIpjE\nMvDErptp/BArKMSIllGsh5ByFtno/qeCzTcFwJ4x8opWxOI8BmPx+iapAz+K\n/Ewd\r\n=Ot+T\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20201212.7d972c3.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot (.blu) file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20201212.7d972c3","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20201212.7d972c3"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20201212.7d972c3_1607756962261_0.5853499065298038","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210124.468c75a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210124.468c75a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210124.468c75a","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"f4c1e23db475791c75ead4641ae7ee612ac6d960","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210124.468c75a.tgz","fileCount":33,"integrity":"sha512-93jIwMgIRm58Ix3J8vRgI62E7W8u4826jhqGmZgMqBI0RJGIcr4WBmpx4SWi59W+G4EdOlze47HIyYBf6GN4NQ==","signatures":[{"sig":"MEYCIQCk2CfMgK5fxqcEqBu8PFCPRrOMq6YncTseirLJwZTv/AIhANgqMq3UIwahvTd7z6PYU7yxqQ1de0QXWbX1hlh0XBVy","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgDRz8CRA9TVsSAnZWagAAJX8P/3TUWgK+GyyRK2m4QIF3\n/UoYRrpMBg9D+R25h+01JsT+0TZJXgBlToF0ygtgnf0DBv2dVjnedak4fVcS\nyE8NT9VPu2n/SZcbaq3WLr7Q/8RCACqzz8V7zcvxX+pk9YBk3CVtxlEIbumv\nmsLyLElItWJB0P1QHDo6uMYYx45SGkyNXRh5HcPQCSbUGbrx1To2KsG1L94h\n9WPizUOSB/f/FDs3/1zrvihLAE+cOsg9T0+oxlJmbasZ21DnpnTIbwKZHYrp\nzOg6TWBcRR9GpPCWLqUmod4gcVueJIM7joaIG77MLVRwCeyHEvtS1P+U8GqQ\nXh/aKGALn12r2P+sTanaMboTBZpeXO8U6KRFQgdhzSVlZXaoj2xcc1zTbvBz\np23MH/2RESeBwBBmQQeYUdkIOC2yPO+u7ulZiVSMQOeLtvK9008Pm7YRDwRa\njk/lYhfFcAO4ysZ0YvAytsY983iOlNDOK9otjiMFVneg3HPnd0tXYfC8BqMf\nkpepyoPkvdqiqrb9zyAO2gHLeZesXnppNVomfFrBoE4yZyDifmtQ5iccyDFI\nVW/vZNpQmOC7PR5MbWE7MCs2BZHy9IKTpHKNOJaBFRL/AGcjwEb4JTXsggWE\nl2/hY8FSIgMS9Lor1AFHUiCRqjmSz0G1wYkyoHRcWA2mpfCRAVmjeqflrNei\nJhJ2\r\n=yXo9\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210124.468c75a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot (.blu) file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210124.468c75a","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210124.468c75a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210124.468c75a_1611472123659_0.7950447847165585","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210127.476c841":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210127.476c841","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210127.476c841","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"d50b52b838e1adfb0d0eee0dc5b6c199d87ce04a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210127.476c841.tgz","fileCount":33,"integrity":"sha512-QUDSOm74uhVPSf/7OZ8nrcdiNMMtNFAVdtFm2ny/kPuQzCLMe4zGz1ER6T9GGluZDn6LZHuY3w2+RQ1Ml7LBRA==","signatures":[{"sig":"MEUCIQDGarzf8DBJmGzipFLXpzU0yCqSJiUjOrhxf2MmNxRaZQIgfTwET3+poPt6yWcBpObsWmvJSwYNu8fO6E0yHjpPwL0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128673,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgERG3CRA9TVsSAnZWagAAAb8QAKGW5U4Ys/Fs2nWVJbJO\nNcs26S7kp5nzFK//uWKeY7K/3MaG7kJJCI/eipQoWkihWNPTBxWdV8m3q9N0\nGsymNSR/RsUqaL9VZT/byLeP1D6imJOJYwUoY3em7CWgrFsKcjQ2v1vfM4WU\nvk6wT29mvpN9fbq4+57LQFjn3Txw33KO8OPIne2J8wZ8h0JJ6XsGgKopkUf2\nEyP4We+I0KyCGgrEI1gr0qnEAjD7qA7zqP1FcKKviy3jCSwe4oEfDyUdTaNm\nDpR5HmMGjPPJPclFAgZFUNGf4xwoaKszj4vC6Ql+1qYWOnKO562jEIgL0AY0\nDMMgdQ/d8Os17DGoaKx2ENTbp8M5eIBbZQEkYPtuma6CMKWPs992Hzprdrh/\nWW43UozxSzEkfjsC+u8irMnrSr4C+wVYHzlPVQZINyQEc1jhVvFMuT+GcbgW\nj7lpON3xDq7pU6++sCtj8DYbvrLG1JprpDYe2C+JYqPBorcPp2nG8ILN9GtH\nSiPiD6VGJxy8WwSTIFiV5qeukL9MTXgngEMsAdlpVJBA28dCBf5Xz7FzhEWF\nFassBUwd82TWuAYW+Eenq1IUHdy/335bMr5v2/PQqnF0F1dw7ehoXX5Bmhts\nE0j7ZFkDD9ov+E/OOCTtzfLwRa8qAq2SvYVGH8mSDDiBJ/fAZNh7IEixpXLS\n+BP7\r\n=BaDR\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210127.476c841.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot (.blu) file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current working directory.\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:build \r\n $ bf orchestrator:build --in ./path/to/lufile/or/folder/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:build --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210127.476c841","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210127.476c841"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210127.476c841_1611731383200_0.3460319706823314","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210128.3d35ae9":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210128.3d35ae9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210128.3d35ae9","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"daec6a63b815fb47886799789c73360537293ae6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210128.3d35ae9.tgz","fileCount":30,"integrity":"sha512-A7ue3uqj+WScLN7iE5fRDkeVAItt55gd5mBI8XKK8rMXeood1nMtHwGAFq6vFRxY9f5r8lbqB/cbV1lXGUlgcg==","signatures":[{"sig":"MEQCIG07K988/hBNokvY6Mv94ouo5s2bxQaFYiUJSMxNur48AiBQ49Z7PSlozsc0W8p0ld9NqoAsbfwP5LkUxy3FS0YsQw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119805,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgEmMgCRA9TVsSAnZWagAAkiUP/jxOwRVpDuu10YcgMur6\n0yF05tQTipLymFmrRPIxdh7VocbDO2CrvSrLhRsZs+j2EmYfRrS4LaeO8U1/\nKzSoqKVocsqc/hNVGGhmIU1NLJdU0CDdLkIZQk95iD06rVRoA3/Z4WQeEC9q\n2wBoRTSjCWwH6nRr1TIzxrv79KW7058g8W/5z+AuxT8V5hv9TcLmc8rFh2iF\ni/qlfXtm/yDnY/aZy+kmnwN7kcNcC2FMJiwzBGP8HW0WKlq3/7rkHI1Rd4Xi\ndR9twAmcmqvUUfYz0OvyBzBmo85Zy+tl+hbEloPl6tQ3lKw7159JwW26RnwS\nu+cYRzBRmA6e3oNl1KfL09uSaneD+jvRwVfUNjO0pcGDze4Ph+Mn2bGzkj4w\nc4S+PUwgmUtKuio4VQTBaOntO+/Ow7+duoSETgRQZ2DBTDwMPCa2txeoo1OS\nhEcfkpYCFx/BZnaheNKMcWc615AC9eFnBMBtuvxoArfNBvB25OYzlVMfe84R\nLkY2CBPZWMRkgC4rvg4J8E8iFoRuVtMJgnuAK03OstfzSYeWejLc1mUeE/Xm\nML0p8Zj5+9cH5+Mp4E5uR70UXx5ZPmYWHl1n3qX+LYdrILqE6zZSqsQ8WO/n\n1H5VTSrYxADqGdhjOSERqTThuWSNLvgzYdEaVq5P1j4i/KjcNm5KohBrT0VK\nLig2\r\n=kpRI\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210128.3d35ae9.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210128.3d35ae9","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210128.3d35ae9"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210128.3d35ae9_1611817759754_0.3232403038636271","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210129.453a0f1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210129.453a0f1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210129.453a0f1","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"dist":{"shasum":"65f54c538fadcb42ef16c233b7bafbf5d4b9e944","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210129.453a0f1.tgz","fileCount":30,"integrity":"sha512-BllO2wndWsk4zfAdpkwt2ZaWf4YZ5dWVMPkGza4u0TIUxrPkHTZRpSUrwcmF6/Kcj74KNO5t5mnbNmTf7q5SbA==","signatures":[{"sig":"MEUCIBLYYGbR7CKdRBoEA00dy76XkA9QqBC5zZ0zpReg4eEGAiEA7gM8l9Eu/hi7ImesycGQP8rBs0L63zoWZJ/pwk3a/7w=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119805,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgE7SUCRA9TVsSAnZWagAAIkAP/iwsn0scSxm/GBX5v0Ql\nN6m6lhXNFch1U7KQVZ99W2axJGwLwVUyYZN3wA21mKZEmI2GiMmrt8p1SIA0\nLf4OHkHKLzKZfP6YB8PWc7PJNpG2FJmIg/QwFSn4jcwl06vWfLApXClar5Cl\nA9yc2yB2s4jaQri9TRovXL0j9hrQym5JOphYuUeHriaczRWYaSsFe1bj/mcD\n8A5YxuD2cg/ePeKozWLLYUUGu3DuamiwIRyvU8OLuABNHOKFZUUszs4+h7wm\n98NbI4AZt0YGSK178Wzrexu1DAtwRnUyEkeVYin2n51OWpGpGAq9YsMAd2Rb\nsZbQ8LozNdUAvFcBX8MjnVQP/Jza+DGkgDQvUjQjGMxeTTRj7++w7qVyroQ+\nMOrBVmN03/oaw2aavH241x6g3clgBa/nseWPKd4Uu6hN+Xbx8zRhV5EC3l/Q\n0wv5PdUlCWd8b/RdyJvteQHaSxiI0mk5606ccGpCXZ3lo2UNPkyZynO0LgNh\nYRNGiDo4b+ZkaUxwbwHKd3ax7Tk3nLG1LZaBsypte4TTx9KNF42kiDeJhc+U\n6CzJHe/4fvgLkgYNXbBgscaP1PRURiaGnZhsmNznr/dKdq/fykRSZPVz63U8\n+wfo9JQVx3dYKRP8vMe+FgjCyDNhUhGvVy9Rb7GUDUDIXr7HVWZBPfcmX28C\nmBks\r\n=J+9O\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210129.453a0f1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210129.453a0f1","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210129.453a0f1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210129.453a0f1_1611904148237_0.25093799655784554","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210130.46cecc7":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210130.46cecc7","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210130.46cecc7","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"412073f502e7ed4cc095e5561e043f2ef6ab04ca","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210130.46cecc7.tgz","fileCount":30,"integrity":"sha512-3uPhSJN1SmQ8IWuK9Xa86fvMNm5a0fZg5crxbHTn+8kx40623kjk8C7Zku0eZjQdrxEyroxGZAFrVG07Z7kXaA==","signatures":[{"sig":"MEYCIQDYeIQPs5cRmoF2jZKRPl9NIRPJMeADtCvsYxZaWQzlaQIhAPt33NDexh5ZI7b1w2htOTz9LEvbEHYomWGCG+bDEsWN","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119883,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgFQX0CRA9TVsSAnZWagAAImgP/0LWGPPKqtOMno3GaNrE\neN1jS6FUCUAxb97fnIk/gZIn4AOhsFO+93ARguLaQj1seTrSHhxklsAmGgXZ\n4CjVF5NNGnEWHM/xeNqeyKU+qTeZoyaWr2GRChMMu7pYJlYzPFetCImq3USZ\nlduwHb6LNbYN4Kh+2QeQ61LVmtG/lFCsEGoCLLWg9hhkBbiXR9MvFrOkTUxZ\ng6IWrk3l/tt64JdmJsarqBKiD/NTRqfjuxSwMWKSBdfJv2exejLljYcJxAFL\nE17Qfka8UgLNWHSahmgQYHR4Y3ltNfSwUC81GlefxtfjeZ4Xr6az+8+yjo3F\nhvLbtTpE3BuzkHu3g8aCWmn7rNHfKo/mGl2tFe8GUxi6Hs6gHAqSYzrMHNTp\nDehKMuBCnE/HOHmh7OLZqfp2HB7u5KShkjo5SwUrTzvUlFlRFxvdgNuby64R\nVx9/dxuF79EfLe9OsVH7r+vUbC/LUm+lpoElhMZvXdguj2X1fSu1Y7pVS3w+\n8TSOmoJzrbxoFPK5kkfQfzOcOX3AESBEjMwlV+vsuhZQVRE9vLl+aMxlhn6C\nfXhHfOReAkWanZ/cLwpUx5P707HRgkt6C9eY4BgpXhX9Bu5i5PqKIm5tyc5I\nylsTGsOgBon4fsyo1dApfVam6VyShYCtaRtifYYdVRsyp7lGN2bTiYRMC+Xy\nvIbD\r\n=0EbA\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210130.46cecc7.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210130.46cecc7","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210130.46cecc7"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210130.46cecc7_1611990516413_0.9858353729509095","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210202.f5bfbad":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210202.f5bfbad","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210202.f5bfbad","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"b135b7592edaaedcdc2263262acff51d8bdcf12d","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210202.f5bfbad.tgz","fileCount":30,"integrity":"sha512-CHN1p3rHG3Lh3bEpdTK34FTcsORYG+meRRoZ9DUoSR1MPpgy7lwPZz7XlJDpX56B1phOcAAOFVAkMHdrlomzCg==","signatures":[{"sig":"MEQCIHUWP9Ik4yjY56Lg+UCuXFkyqq3oM39HCXyDkiZ7OdG3AiAjtoz1kt7L/2Iezx8kwRZ0oT9XHqlFdrY658sxqC1Ozw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119883,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgGPqVCRA9TVsSAnZWagAAqpEP/A4c2swQpCubOU41klLS\n+dSIZmT8X3hlNLIuG+cJkgVE8lNRzDcid4wJQx32ZQWvfdcIrA2OzuV3W6La\np9XpDRYr5NsDHcL952zfznXwsZXtX3siU6tZ6dVOUrAdW7n9fSbgnJHNB0Jv\n6wrDq/Mvpc2twPU6xEcOjEhDroGLfs3rA5QaHer94sb3hsyG79vCELcwnPZU\nsh+JM7wdBYeDc8vQ6LBOPvlNBzWmB84Qxf+cKGTObJTyFVa9QAoGmgsZDSbD\n4TaXupbIc5z0qxXpn/FewpFibp0Ye2eG9ZAoM6K26zqsL+K906r5OgsQjR6W\nQdDREdJQHNElMMDrgaJ7TGn2ewOPZ/1HS3+P9ne415ocsG057Ge+PLtkod1w\nixXAJNJ2KZM7BK9LF23Md8NbRCS2EsIoInubpSUDLRc6PEvTa1jdvysJfDvT\nqc0b4P1obMwAL5DAegPGlWl/GKKM23T+46RE9VnnQN2PB4vzym9efK0SwzQz\nG+io+WImgfknW7Y19/XS1LVrIL0uBsnz+bcUiRjS0vvYGnLzrqyMkHzBS4v8\n64h4IP27jrlWVupEKuZREbKIlGgGZsdd9QOroahVWCb4K499BsDQGDTdMWCm\nSXk+X2lwsR1frvf0EggjFDcEx0ebvUz1465xPFhPa29A8prUpaVVzH2dYO2k\n4oiS\r\n=BKhR\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210202.f5bfbad.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210202.f5bfbad","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210202.f5bfbad"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210202.f5bfbad_1612249748794_0.9067395363165613","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210203.b0a8b02":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210203.b0a8b02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210203.b0a8b02","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"d0ef3406363d70320d4f2165c1b8e5f54c88c756","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210203.b0a8b02.tgz","fileCount":30,"integrity":"sha512-GmJg0Ryuxn2Ek8w/Uf30ulBQsQ+HZCzL5gCQckmSAgJ+WLubN93WgnNl0D6MybH3Xqew9xgf2fnkmyaizf35uw==","signatures":[{"sig":"MEUCIQC5RAZMn9+ZROy721jwMcUZx2SOdsbuKJ9v2Qt5uBzp5gIgJBoju2VzYA0I+/qKMt3qhWH+vUDVoS8atYFlcJphD5I=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":119883,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgGkvCCRA9TVsSAnZWagAAyusP/115tJgXpaVjTUkgIVLv\npPYv/tyHcCDo9b0f3h9dvVKu7dsQ9C43OSqfa6tlsjMocXS7iun/QISSFAfj\nT1Nm+FQU+2cQGD4gP8MrEBGcq2r6PZ3CZJcsb1bLetSzkMNqR1CaKKp4u752\nJPLVwDJ6OX52oAFGyLYBa8AQMk7mOOmYxcD9mgCyI23bsvlTtS7Lq7C2ltFG\nmNSMbJKWkRRCzS0E8ksojIt4xXj4YIN+2Vvr7dCCkO+trywRi7BGzgvEk7wL\n0D1bkpZ5G4eiavXo15ZuPwfKVSXRfqVezqxCL3+iTpQ3lFBUWH3HcDM/Bg0Q\nNQL53s8G7bJ+jP1DmFCvu8FSczA65hRrZu5ZoZFT8vd0wYxNGcxjMam8EMgK\nm/cEUx5hBMFZGv22MpBDgshV+EyNirUVwvsAKBIu3XuLGxTZNF98DPs1uHhB\nXGr/PqD48+0mHo3KsLUBpHgQt4BizF/is0mk6T5rvJ212DYa3i1TVpoGpR4U\nJbvhFaUpZeL9yNdm1FLompklZcaPLe7EopLi+2EPGIPhV88XXyzWDYRWO2Wd\n67D5Ta0SIKiL6m8nP6oVIr77Xa/mQ4ONnPGjwwx0XFLmnfXcwJekCNVRsIIo\nptfguNIZj3e5wIPkEDqj47rHs4J5AK8PdFjGJcfDPks38+a21Dpt8Ku2gwgw\na3tW\r\n=Gx+B\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210203.b0a8b02.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will be created from. Default\r\n to the current working directory. Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed. Default to current working\r\n directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=[--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n \r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.8","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.1","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210203.b0a8b02","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210203.b0a8b02"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210203.b0a8b02_1612336066153_0.44164374819812013","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210205.12121c8":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210205.12121c8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210205.12121c8","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"3c4ba53d3f4bb178d27ca422d1738aa99b42db17","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210205.12121c8.tgz","fileCount":33,"integrity":"sha512-t51GlKrmh84KouvrahHsrZLjsdN8s3+bgvysZkg5Ik/Z6OcQ7GZHVxcZVCZnBb3oTdcSA1tjJtijrU8qkb+J9g==","signatures":[{"sig":"MEUCIQDQeW4NmdDSEkLTM60jjYApZ0CxmHG0oZQEMsmIla/z+gIgbxIlGPMLbL/8UlAgmfWnay7k4/jh3lv7qMWeXxATx3c=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":129056,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgHO7VCRA9TVsSAnZWagAAfPQP/02ZerhEvN37osqLeR4V\nw+EdzaFAKw9dlD1NEs3CB80lzwtRISJBIVapA3xDZXv63uVwIQAsIFZpdSLB\njaKTLek7FDK8E6PKNDcaRBobCgElPKqk+YgNAxl7JqVwcGQiwDHkxDUnt1rd\nAJKyk5KyUPeZ/CUXzZ+ovpI5AYG/2ZQKbPZKW/7br7//z/qd1w+cWYiz71w9\nUB3ateqJqnXlmPHNOKZYTJobaS+ZZxr3ljaUk+iT0jfluCg6So4xfoiaAr+y\njAZr9J0EJ4894OJ8/BDy9k1kr2lQzHankEcSHCnqsCcX/ZipsZTH6XJLEzw5\nJIrLF4/UJ/rZd7xplF901oBV///xT5/Ln6nWPKIXkPhD8+Yo2JiqWn+Zh5Q2\nVbbapDDFCxyFpBpPCly12Kbwz0NT5sv5gX0xXVUbW3B/rKs6Y22uSCEPkh69\nNAhYr9RPmUqcVzkOY/qOAswM6S2PH8PRNBzE+Pj/CChUp7zuf9Bsm0wOoMU3\nqceLlF4CsIgJT2ANE/d5mN0asr/a0WbXW/BRIxjTtLSvpSQWDc7A3NUfI4sh\nRn8+3/gp+/gLnxoSJcsvsVwjSdpiW5VbGwJSqpXNuhsHM/m3a/1vn5DxItSC\niIlMEnnn/gvhgNstyVWeOIy9jM0ubljwB32WZFXqpjDVbrvWcZi5hTd0NE1D\njhY0\r\n=x2+H\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210205.12121c8.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210205.12121c8","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210205.12121c8"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210205.12121c8_1612508884859_0.2665938379616488","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210206.dde4bda":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210206.dde4bda","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210206.dde4bda","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"712bb5cb0d153df84f8f52d93dddf0737c7ea643","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210206.dde4bda.tgz","fileCount":33,"integrity":"sha512-HQWhdo7DC7O61YmXGJeXPgt5jsdSuS/P2yBsRfR2+GLz4yk1t0Tn2gyAQVj7IyLU2KdJFjbqi02MwVYp3PVrSQ==","signatures":[{"sig":"MEUCIQCWgfETfu4gUXhQ2So0gRpAEpmDq79r5yGejXNCQJaJzQIgMJoRKaVaIGc7RBCTWE289TU7gFrYGUL/rQK7naTkYL8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":135074,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgHkCECRA9TVsSAnZWagAA1YsP+gP2JvhdzrT+cZ3DPdoQ\n3miDZTHwAHbuk0qhKX+clQxNXQ7BG50fcXeKxKc++U6QiDP+d0Pp7BGj9efV\nSrPAgANHvN6QHf2QZppH7yDU7HGlzZo0U9339Hnd82g0OGxUKLaAywLqP6g2\n5oMzKP8IuJWYP6UOi2Z0z4y5AhdjAkd4OvHnfAB8PAgN3x/+8yPU+6W7Nv/j\npV3hacCJquBORlCt11etZlQZcDIUVaX9oCvhyRw2C3Akc+nuv3c7xyjdoWbR\nqBd2E6ud2cF5RSTBqydXw2xakKClr0JRkIt6E7cXeS4O6JzQU9cpVLoH18zD\nD9cFRkBhANwftaUMcHIQg0Phmjiovciudjyi+AX+tQcJHgQbXMmmRnseh/5+\nqz3M/1Y5yk5nZ2FjGDOD3RMoYnrfzi9t7vfzJHN8AdNst1M3aLPFB9grCfwv\ntmdlX4aD2h/JYvlNofsi+K/dtCBBa6SewBShwcypGhk5ElJofd04Nw6PUzZ0\nMkJ0+HnnDESvW1T/3VITnw6ATIML5EUvejrqR2ydguVZ1dRwvg3ntInspC/Z\nRmDIQ6hb7y2xBMc1/wsCNQ7b+lq+Om77nzUGm07DmbyWPhzWykZYXyBGBG6f\npPVO+tXJvH2lqMt0ia7oV+2jPBxo8zgW8jrrKOp/26qCiV1Bir4Gg6KxKDNw\nVYk1\r\n=RVYQ\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210206.dde4bda.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210206.dde4bda","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210206.dde4bda"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210206.dde4bda_1612595331948_0.5132012505169989","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210207.af5cc18":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210207.af5cc18","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210207.af5cc18","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"08cc9d2bdf8c02e4692e118794f2315b02ae1bd1","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210207.af5cc18.tgz","fileCount":33,"integrity":"sha512-km/+Yap8kZud8ODURZ1pUYaww11QtSsLxJbqYGyIZyyUQ0j19iCnxpszdnXPbTlXGBJwBi5UqzY05d1UALJhPg==","signatures":[{"sig":"MEUCIFskwmftxNKrAKh5AtBtFghpfQUyx1XBtJr673uqnvX1AiEA01RKlgaFn9TD4SapDpFm5YxlYR3DPSaWcZ7r+TUd1P4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":135074,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgH5H+CRA9TVsSAnZWagAAKAIP/iJj4j3zqlKJAjQQtf5k\nKI5b7lV9tckww/G6HziRWQtI7nsbqD/LiawWxxnFXZ9TGjBUgsgTQyLVvzvk\nKsK/whjqWDAQyXdx3fBgRgUPsjxqgrgi3Q9Sgozzdvz0qpAOjIETekkT9u3G\nRhgtoeFsvCpZnWhbaE+zQ1NZpNpHXgxZd+9nwWMXe2YOB8Dl9UmKoUyBr6An\n5g287woPDKkEUweouDOKj+Fr6lE+bE19BPgiU9808nU42ULdlYz+1j1yqk2w\nMM+hmE8NvTfYjHZ4secL6+DQFkfQ34wZcjZK3mjyCFzn7aQ4V9pXrpGTveQ0\ny6VZer2ot8IM01kabHNquQ35CE+tazLqZgc71dRbXRHexqV5hdVaXmBjwwIr\n/aysWFttwXdW+a9g2L+XYGdGJk0qOixneZ8oTRNszn4mjjgdO7/PhApzy1f1\ntO8iY4IVBLoFgMmFJGuF2Ky63yOKH7uRf5EU573Buqmh2/AQkHdG9WSHnrU4\nFbhrwqsOwl0TG6J1vkXbqJXT1/3bEHqnsilOA7zJeJ6UAn820FIMDVP+nGzX\nld5vjpxkQVvICM4AvAqqRCE+1abBDqTowm/lm6uYqmhpbtIUQIq8XD3KfNEq\nm7gLOljH8xKGT2+dbI9icHniNmGcEsy4YH4I0ghY5iVcCY8Xy/REQocVENfd\n8/aF\r\n=/Q+r\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210207.af5cc18.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210207.af5cc18","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210207.af5cc18"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210207.af5cc18_1612681725903_0.6083841754658466","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210209.a44ee5e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210209.a44ee5e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210209.a44ee5e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"586cbf2771b8ff247e98a025cbbecccf9b7d063f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210209.a44ee5e.tgz","fileCount":33,"integrity":"sha512-aY5FMrJzV1ZY+uVxZcj+GOB139D8YEuKcsHeHggsah3R1JARSx+SRCvsMaUhOIdNvxjsB4qovYddL8cd8vTMFg==","signatures":[{"sig":"MEUCIFU4b4aS2oDavbGLJB81VXOHMW8rZGSwrsGzpPEEemgIAiEA3/I67pz1E/GzQ8pjuAZtS8BGcdaAzezBDeqNwYqRgfY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":136030,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgIjVACRA9TVsSAnZWagAAVJsP/11p7tmH1+24/0AN5bnD\neeT3/T49MWqTs5mzms49BQojbhsMJLQGLk017MH4dRFAPpP9PgszXEdRj5gk\noM3h4ej5LcZ88BGgNnWFn7ffuWnERUg5Hi4Ie7kB2M8jXo2W4wxeSqVqiJtc\nWMHskl2mnC6ASEV79xgonMRjjgd4z4w1k98H70iWdhYm7snYMSRec7Wx4EPd\nRnyTe75e4x+ZQvz6q0oVVqnILArJY+v4gt9aCI0o9eemseA9B6R0SUJP6P+u\nY99wrO3GATqtsl/bmKXa3gCBWzmjl4iYcL7huf4FGx4g+1abFnH8ojsuQdYv\n/gnyt9oQxSiqYmwuPm4OoWBk+1C9K+Iw3h5/avQRaYsKSRzCv2Zs/ddlPBP+\n1SCMnOhq5PgG66WLdbPN0+3mRiCOw5ojw2WN3WP5XtG0xT0BT4LvMWe3Nqbr\ntWsWUPSgp91Cwb3iUzDmRxH+ipo4TiOJAn5u5nYi7penHwyzAov21oK7U9kw\nP4cGF2O5KJVSvDl/5QZGfyzcZCZaVI4K9edcTgvaRV2Ziechv46v8sWK2td8\ncsdcxzhBu6J17tAtWXWS6TEbtPPZgsQfe+RuRPzKuQkTMfGadeTOMD3ICsRI\nHQwI6FFmDVda6sUo1wS4zkiVcQwGLUTes5w/JVG4fTFu350WDFIgHGE2P7ga\nNzfd\r\n=hg1k\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210209.a44ee5e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210209.a44ee5e","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210209.a44ee5e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210209.a44ee5e_1612854590170_0.07955857354511187","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210210.116dcad":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210210.116dcad","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210210.116dcad","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"40a730396178aa24eaa7c9c8ca263a6733076fad","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210210.116dcad.tgz","fileCount":33,"integrity":"sha512-ugbKnEew0PxPOf0Tb0cB7XR/CnKD9uvgRMN0JU5xY6Yzqs7h9FSELdbBHypmcemhPk1ATVfM9JI7d2HQnyunrg==","signatures":[{"sig":"MEUCIB8j2ESjERvhAzsejclNfxkOOq3pHlydPEdUyv44v7JIAiEAo8WYg0HVnS9h8nlLUodlyGfg9xWTVTpSWYqmaRE210c=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":136030,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgI4bLCRA9TVsSAnZWagAAMN8P/R3JkOeVyj9NtnpQVpna\n17O1JKvtAq+3xL7jMVeW07OFxhNhriCaeQyvM1R2mrD7KQX/7Ra2czJamlqM\nS8n+nkHVvwcv5TGq/VFWUILsdxTbIfMSMDAJbsiXMYdYqIFH/z5DgfynB+sM\nGg3KLg0iAqcmS1H5AOzhtvf6H67mVEgm2m4sEE3Zv7cWgRqNAR8+bHVOJrIH\nYXvfp/0KLVCGGlRSnp22SI4Z0V8xnFjrp8yn1EDMOPaEdGflXYLm4cqMl36C\nXkcVTZq3sxt/AgcecBr1wgmDYiuPTHH3UXkNmsAkN3afoZJf04KXDHr5oZKv\neHefaEV+dKXTXph8Ehibq3g70++9JETlbdW0vdSxdKOjoOzEssAZcN67Ktm3\n/haI/Ru13VHNl3cHZ36De11JEzIhzgZ0v6qx2WRPLNj4NO5owlxlmrjw95CM\nPbiGNgeczZnntC8ECOlZOU86G6Gehzo/KJ77fw5FYlaG24oTTixz9I+z6f4N\nM/TcUeVDu9UKHY56xTeaWuO5iRcfpqsvEBQLa5BimcgHqi+MJiFrgcfZjb16\nBPLdoA7405IZ9ipIzhcyQwcQmyVIpSxq3R4wi7rL8HtlvL7g9jclco+jvKQF\nwZf4M/chbUEpm3i4tGtlGw+EEgum6j07bStM8kzSObgH5v+hq1kvT5G4boV5\nsMut\r\n=Bqcl\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210210.116dcad.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterancce Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterancce Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterancce Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterancce Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210210.116dcad","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210210.116dcad"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210210.116dcad_1612941003102_0.9690869421891193","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210211.c895ecf":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210211.c895ecf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210211.c895ecf","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"9a0177b086a16fda9552a818bfb28797e7fa196f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210211.c895ecf.tgz","fileCount":33,"integrity":"sha512-raRaWGte0zIsfCC/IPs3dF5WCpEmDPn+57x839RYcGtBbB9vLQtuHOhFKKBnZAx2kdDRcij0X2tYThQ3yD0Zmw==","signatures":[{"sig":"MEQCIBg28zHi0+T6G375OueLmzacAsDLMclgYmLgLxouXtBAAiAVKg5+7wbcIsBFInQ9YCaAx3/7Dxlw5ZQc8jOMYweeWw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":146071,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJNgzCRA9TVsSAnZWagAAmPAP/AxILB7fKnzbnBc8stbn\n38QJD4UhTRYHG3yZM7qK5pPV+jgDvR3tvb9QF2LUseQxbQPNAAW9m4sGR1OK\nvA1HXCeV0urROpBqsYphvU2xkbRlq+YsFwYvrv3n6+6XBzOX2fN1kOzDVLHZ\nGKb4NLZ8OGwlN98csT4OO2ggVl9bOeZyjzhE3p7EibtWCqKHyncJGreJ3CxU\n3iPwA28Nc1Szny1humymUgkNrKuFXsKvhcxKc9SgINil62RI7jRNRnXUcgCG\ndAlMACA+4sDJMsqzzuLfVtb80pA0dSIeXji58JnW7imWjwDX1yPd7UDrkzey\n8fLkOXZxKhw7hMBIb/WmwIjI3Phgk9I38uUJCB+8DANG+TrDQ+GAYRIXOe+i\n40T0MLZpowlzoclTXg7Gi4aAWBy+sn3EXluEw5fzFUwYTutu9LB/a1ovuQxS\nu1bcMQfMlvvvNgNaVUTXzWWHvjSeCkoALa5NgQCizrrPo4s7wAnNgh9qc0fa\nkOXmYbCPv0y/PdyH7JqK68JPZYKlr7rsay3Gj/TSG1V4sQzmaUEKDCAY7do3\no0M52/JodBEGuASJ9Yn65RbhfC77fYtTeuIeV7T15eA0YesmKDfISuRRdGdn\nXAias4IPlh6TzTBSPCTYpLE8sCzaakyHKjfRkOzm2hr6PEAq2pS8bJ56i7wi\nLLwv\r\n=VUDM\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210211.c895ecf.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210211.c895ecf","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210211.c895ecf"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210211.c895ecf_1613027378732_0.1982476123529675","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210212.2884eed":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210212.2884eed","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210212.2884eed","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"e1f086373cc85b7ba70827b938cbef42676b1ff7","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210212.2884eed.tgz","fileCount":36,"integrity":"sha512-m/b3vbZQiIaK/pAgrfkZTYiL9gOw8zK5467xCFQA84DTAo41VX77gPwGCznVf0/S4n0ZQnbOfJzQhdtjJou8Iw==","signatures":[{"sig":"MEUCIFJ7WfKuXpeOlrgmw6CrX+h7ikuLihuQ6ynm9f4lw6mFAiEA/WY6iOjIYE+HyIyTNhleGYxwpGW/4iDPaCUQikE0b6U=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":157127,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJimqCRA9TVsSAnZWagAANcQP/RfFA4QCyfAwLjssTx3l\nm/GCnon0jhebqQ5oQlo4QS80kLf0wt+amr92egisfoMAfgrZnPx/aCXdsjUM\nTtVSvkSGYRiV5sUECHZd50/wq/9m7N90C6GJkAShAX6QnVm9/eEmM7dDyJIn\niE33iEtTbJKABZFyUCX42Sc0UTzqcoSaM5UmlGrRn+zFSihz/Dfi1pnIptvQ\ngSc0uQiCSdKJysK99XSdj9qELXuhzR6N/4qkNZHhXWxeT/zunoZ8L+FxY8c5\n3r3fAH8dwc43TYUuTBp8XzF5G2KHDEABeBHfGEOwMux9SkPdq6r1fRS4Lckb\nNWHWS//sHbMAu/NagFqdYOO44ojsbjnY9qDRYWhZGcVqbERUDgXJAnrIx4VD\ny2VxgHHMBahzRD/Q5es0RUjAryWIy/itgbwiDL2rrOrfs4dZii3xGj8fuqW4\niiHxDXBBCPmlmlyciAp8RtEAX557sXjGOOkR1rGDg09Y5fdDhBec0icUr8UR\neOv8brCAAfHCAO55h4pLSRne+/6BUovcfd7jUUYH7roHMEjrkVvDlQ20dHpj\ngNkmM7MpzJ7EQZ9nmUl/ucyF15iuKgAykjLY1/i69cPTmlB0IRpC3AqNv8gt\nqsHAvOwJQk+CkSGRzKHAPlRePEFpoXK07VBc4rpovES1xJQRN/rG1qGYZ0iC\n+Ep4\r\n=e0ld\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210212.2884eed.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210212.2884eed","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210212.2884eed"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210212.2884eed_1613113770193_0.6592914180200431","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210213.4729245":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210213.4729245","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210213.4729245","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"37bbf8bd569f9947f0647f79aea2360329ecc5bd","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210213.4729245.tgz","fileCount":36,"integrity":"sha512-Hlig6x2K3QJtFHU/eG/24ZimITEpL+0998OvZ+MTj87Duh3Q6a33dyEUPJi6liCzfW4bHPG1rRoISS03S8Whvw==","signatures":[{"sig":"MEUCIDeRgwh+MMLWzsemeJcy45eCRKRduagjNiD0CX4Uk6NyAiEA/5D7cJk03xVRvLKyEp4RpUQmTOWTmvu37pXy+bItrEI=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":157427,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgJ3tCCRA9TVsSAnZWagAACy0P/jLh5LPGQKa+5MaUAmjW\nkMW38M/7Vsjvyg04RLKYQaH0tmk2XQcDEKHgl6zkAVVsjUfBFG0pi3zpqKtO\neL47sQYdXW54WolOfW4R7ShejUpECzrc1mvZJ3LrA5uER+CPEZ2VrLecIvrP\n/cWXjPw5QyJbmoXbMP3+QwROrTXhtXVr9PuXuSuXntJkljiI5v2QR/x3ksgq\nsq/TKSF8VHdwYM1X0y8w89302X+9Ao6Pr04cdrxEVXlPTdWgIzixg18tsN2H\nSfLzeLdduVR8hA82KTwp7egNiNdrnV8ycxhxWeBo66ItEay8Lmngh1tg+A7j\nTb4TH25GaPeglJ+S7cHQwFQMimjqxXp3TwjJKPrMSawHkVehkB1I7VkMRSWq\n0HZTqc8MUUkxRhtuFwRUeVqzQphdsvZvcQ6cvI4jLCTjaJFjAi9loHeOjuiG\nxuUUeS1ncG95jnZblg0yEeGby6dxne5ZFWy0A5UazyoLvONN5y+0hHwLZAva\nAv6aU7PB1luDPXxhH2J/QgMrelmlpYA/uUK68ojEosFSZZApJvx9xtl7YAYI\ntkisE1HgfmKROZgd0DVM1h148R8mZkLqLj7WZr6GK+k6p9A5A/b6MVP3onpu\njUgP6Zwmc4eZw62Sy0u3SYIv009QXZNdFWa3Zpt/Ze0eHLhlfZdRE8OyGH82\n/cyV\r\n=8sUC\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210213.4729245.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210213.4729245","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210213.4729245"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210213.4729245_1613200193846_0.058106627082384676","host":"s3://npm-registry-packages"}},"4.11.0-beta.20210214.41d8748":{"name":"@microsoft/bf-orchestrator-cli","version":"4.11.0-beta.20210214.41d8748","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.11.0-beta.20210214.41d8748","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"c009dbfe7e662260b9de1ae548e523210352fe30","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.11.0-beta.20210214.41d8748.tgz","fileCount":36,"integrity":"sha512-5SH33mOyKoh7DmG6/JWpTXRwgBr6BT0h2P+etiYwaqm+AdVUEeUSUboe4NbLwzYeCFh1kdWz9s5arJeIv0WVVQ==","signatures":[{"sig":"MEQCIBAsgnKf+H+dLVOvtiY2iXHH3B91AIHVsWmsxwxT7wi2AiBGkJRf8WhC1QJQTSzH7AzEZGN75iuK1vGpINIA0E7Fbw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":157427,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgKMzFCRA9TVsSAnZWagAApkkP/A+eWhkjCNgSJTnKWYDj\nHgRyDICUm7knPi8Xl26T3g8FBlKOBYZWqpnj5WxwvgGS4h4Fsf1qF4bvlEr2\nUFv6EgRd/6qnIz2neSFrdEC5rEx0wUNFBrk1DEnIq/taL26nI1/S8w9jWOZh\nIE2AIJYXmAitJss7BWbLix0RPYRkgVAIZJwKnkH0dJHLjEU8w/ThncAEe/uc\nvocGPdHexCtLD/DC7AhmGxSBjJYoNLf/NqrhjHF0l6mEaobGReiOEZPUd9S6\nTmRp0jQ0e3/7Vf/ogvc6Iar0DiDTz6WhsGzYCWu/4C/GEPV16XTfypAd4U/R\nEsZZdCh4SLEVDclZdyeiTABgMtyD4PvwvzFXU2wmbiITGgl5JLt/Yw10GNWf\nduCnmlCzaytKEyIxnYoUBZU9yAKyimA+aySPcc9l0oTsJrHtWiy3sr3e02bd\nJVTQCMxfjxQs7O/8Flj5DUZcbaas1RWsXsI0LBFtOPv0vWnq7gtf00vvWK03\nO0Ej3/XAF6+3oX9sErCo5hJ4vojWrakIVfqiQeDk6IfN8q+NKwrGLjvxCwK8\nTHgkiDVmipI6dYvuoVdQHCScs1WSKUEHHRqaiTtJBfgX5r/oTUeK5ttmHcSm\ng6hIPVAtdoToupJ9gbtjtDzfdNfW1aD8N3pbMwrAuIaCtubwJYpjdD8OI8Vy\nYpJ+\r\n=hF9D\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.11.0-beta.20210214.41d8748.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.11.0-beta.20210214.41d8748","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.11.0-beta.20210214.41d8748"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.11.0-beta.20210214.41d8748_1613286596621_0.654148168440547","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210217.668cdac":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210217.668cdac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210217.668cdac","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"05b93da89486cc36b8e21e33dae6453a76561d2e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210217.668cdac.tgz","fileCount":36,"integrity":"sha512-epuxotQpBXG9Ns/CJ9MG4JqjYDDsb0U/FKo1YeC9jAxmwGH0bjURLoBO2pDYnJNkT8gM2T57/C99Ok0TWmM9hg==","signatures":[{"sig":"MEQCIAxC03tjK/A7E4r9UVb9IQrLSIZrQH0msZzrIgiOSh7PAiAGAUni3gvhDn+ZwTL2QkhUjcovPt3Ww8jiaUO6RHrsMA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":157427,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgLMEWCRA9TVsSAnZWagAAWgAP/27g7pNeZ+WOezN9vBfU\n+touGKIs4uulEijgY51hnS3Wxxvu9XttpGFriJ2ce24vuR3lEHZlh9MA1o+k\nqXj57FVIxaQVX9qW86eCltk30XjYvXdMLVn/jxivawNN67Wm+gQUsR1aolVt\nAeTSa4loFt+dBC+KP1GZYOeAvESPXgUExygkniOMH2MAP4iC3b+42Q15o8rx\np7ERt+oZkcckg/cfbCBzb8ddgSgP0XkxpMAJ+VPU9YUOOJr6Z3AXq15Xl3DX\nZyu01S6sOQU3f9VBm03u53wbgSt0iT5xDHeKtGtmoGYJtpREpV79cOJyiqTt\neKciOd4UDA9aG00E//0hihGxKSA1B49Hev6b0WPyu5by3MTdX2djNMTzF2tS\nwggU8J6Wresza6HpZ8QqwX3pgRMbxy62dDtBwucZU/Ao4O+uQ6JU2cTus1jc\nZTAXQHkDRsYoZjB393dAEU8XmtD/l/9Ajt6FebENHZX8HHQQTNGHRVZvxsp1\nHexcMq57N4cmLgpbegcdON2SxyHfHHC0EaZ+L7DHJCI1JswWXadcIeqJSNqb\nXuk7+ghM4pR7j+4ckE3U+H19S9F2p4X+9IW/ypx+/O/209PRscMgd5VgIs8S\nHRszwlYiEGoDjJloVHMjPin1XiCzY4CqoyLvj5mJ16Gg76smDehrfh3Jt4w6\ngkRr\r\n=CZZ3\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210217.668cdac.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210217.668cdac","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210217.668cdac"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210217.668cdac_1613545749797_0.32117080871757575","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210218.668cdac":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210218.668cdac","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210218.668cdac","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5a02d2992fb234d095dd1ad23e2c62f7816151fb","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210218.668cdac.tgz","fileCount":36,"integrity":"sha512-x9OKGMV78TBKsVzXWHIDsiGWdGCjCDDS//LAnAfhQxrCJxIQDVBtbz7zmJCf8Q6Int1pJGz00t2RXRDpITaE2w==","signatures":[{"sig":"MEYCIQCOWHREbqnYHTSn8XI5KLM2/V1rnhfsDmH2QOmFTjQcXwIhAMt11vzGNkzTrJDVe6mxyILNxTF9wLbvRQI6VdM5MDj1","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":157427,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgLbh/CRA9TVsSAnZWagAAvY8P/0rTsizU28YFjiOl+kbi\nITBwRD/2RcGVPKWmUBYF/tVTok9Wo3l0OWbB/0p1rm0W3aO0JVFzjR/y3Q1R\nFQK48a1L3kVwo5XS+ixPAH37bq3NyofEcjEw5zjfSIQ4m7ps+9WlonDSGJGC\nJOLr8nrWalYMTDrjGytAND2NDsTckGdjW2bV9DghmBXEysCpXoQHHJlJbf96\nS5LrkSqrDKRcmwM67zpSOY8tsJ4qjRr7avr+8nLtLvVffCU58ze1yIBbjuED\nqU3E4xarteOJ0W7DE+024F7JfLEzWTpl+pyWB5ByZLJCjFBSIFe3rEIMxhAF\nc4YA084MPLwNOMV8LTjLfJl20T1i2IT66iqcF8h/XBzA++YIM6J8f6v+1OTz\nFuGV2JGUng9hq3cwwaRNROSCHF6ilxpecDrQbm6BnS6D9RE0b17+MtmhDq3K\nlG0jlr9f0zOsXJlRwU4Oav60AYR/JRnEV795TbwKkOHa+inAVMiPj77AeRtr\nX7fk3VIynYJ3L0PLSLBVSUZtzIopOtJ0fGq/TMuT5s8gCtWIqjhCOgZm9Pva\nkdaNNaqUusj5iYnJ8OKYlOUf9z5LYD+GNQxi8OSEtSYQVRPxX5pkHSE3rrsG\nLhaaaewVH688XcTAlt6U0mzoe2OpWallcVk0Zlol7zyqU0Kh/n3h7f9bHJl4\nawEl\r\n=CKwU\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210218.668cdac.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210218.668cdac","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210218.668cdac"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210218.668cdac_1613609087134_0.3810601000358689","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210223.d42fe4c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210223.d42fe4c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210223.d42fe4c","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"7af9d912302c9d9959ebe6f1e9eeb28ac22bfedd","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210223.d42fe4c.tgz","fileCount":30,"integrity":"sha512-QuT/3kvhc40M8EyvEduMvzrs7o+gOws9Rajtr9L/ySvXgyBe1yXKr2xzWxvf2KB8W3t1qq2AVplX08RmnJ/5IQ==","signatures":[{"sig":"MEUCIANj294Jg7khNirHAFMXmIdzM9twHzKDz13iHhbwejvVAiEA/l3D755CQXROB40kAblsdjE3hjYy0vDmV0v3ppKgpJ8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126561,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNVewCRA9TVsSAnZWagAA2GgQAKGpdLeZ/qLxnekTEjZ6\n9jIrzrBTCdIyJtZJPiT7imcYQGFbS0PMGjQiS3qVSKSAw56TFaf9MbA3Kijh\nptdS0wlDra0EsXs2fAAhqKsq73sg28DlaMI12LVOGJwF85rpKO7XH2V33Elr\nusZjC2qdxA29aaOIUJ8HfTfcebEMNI8HxfdmfKRb+D9+QM4LwRhgQ7XVe7LI\n1Sc/B56r3eMS9i2C/TiOmtIDHmTTxa83TkM6xYlu5o3IiHajqs2c98Uk+fpV\nYzsC7DXvBje62k34UO0+qN9ycg0Qd8qSQq2iwnp0bV7h3t37iVuga4wrjLb+\n9BWt4avsvCCtwbDG1Y/tjx97i/vGn7PvhYk7C9XkjoyjmIeb7FYY+cDjPCvq\nOHwQW4d2T/lIn96t8TGEfFbgChtuhgoyB5Jmbsl+D0lmfTfS6LhZe8J8lr/m\nFpCWuo6uKw9s7bCO9MrlXTsOdge9YNSTiqo4O5QYdiwYR+lnw5wEg0qpBNj/\n90fWDXz7oH3igdLh4V9rm9eEpVtw5u4IKaMOic6ZjuVqrN4alAlo7E9N32cU\nJv0PayHKVrwlvqke1O8IFfRMj1PDnb16vmioOBoQBFNTcSh1SJfR7FMX003h\n0TfESrP6pZuTnm1RFlYEn5hGVtp4Igm8eeColdZdl+tkb69h1Q+KVg/0tXh6\ne+VK\r\n=VU9M\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210223.d42fe4c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210223.d42fe4c","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210223.d42fe4c"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210223.d42fe4c_1614108592325_0.15791025066654552","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210224.8610f7b":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210224.8610f7b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210224.8610f7b","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"77d55e8024ce85f0aebba2f3cfc0454e82afcf65","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210224.8610f7b.tgz","fileCount":30,"integrity":"sha512-+xBUJMqzBeIKEV5Xr8QrhNR+MBWzjJnL42Rlc9iGwqZxcgMOiwZaxsjYaeuTqMPkVH5XKkyqsFFw74iPBy84/w==","signatures":[{"sig":"MEYCIQDyUmPTrbrn72bm8y8uoEWnzomqZkYnkkFU/Ksk5UEgIQIhAJqzyHxuYLXFfmRVcZobz9/OwL45ZZIJBH19wfUW7tiv","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126561,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNgSMCRA9TVsSAnZWagAAz30QAJIZRYCKlk3oVqcv9tMT\nKTJtpCgbV9kkT0pzG25olywshn/DzYI5XH9tBgBQ01OAr+eUL61LdM4kK5CC\nSvRK5JTANRo3k+oJTSmnHoaZB1RNdMAh7KP1iddGqR/Lse8+mQklSZufpCRX\nRMeqMw64n2+OFYmHDOClmCHqQYPYodiPxxZXIUj9HAQ4Udk5rxaAk8yAS3Xg\nKw9k4oBcVm7RCcZO8pHfcrNEC7EfuGSJ2D2Tw5DRMl1nktH6vt8w3xRTPbPX\nVBUacMXtSUPJ3vOjsxDckG3GuhziepasyrGo458RT4tW8shsnxhmyppCx1fH\nz9A/zF8QEdnqpwavH4ohUQlA2uhvAaBI6TjKHFRHBb3q1hwFYBYVUH5o/ME1\nGVUsNZA6PZR3HH/eO8l+KM5aEHe64O2AfFRfh5gAHv8FaDjerAV8jX9fOQUb\nhak096IFnV217Ar+mD75p1wjz4v1+Udapkx1AwGFP5Pr63cgGj8FiPHLb3x1\ngUP20FDPAyO8SH2BSAaVR12H9oLod3HfI/qCZ8RZXoizZSjnDfSCTgxYBCVW\nLXWXDdAowhKBA4aCQJREDRbCKbLoeFsdkI1BjxJjDtNl9YyZwEWeTlAozqP5\nU8Oea26M9lLfooDtnt02/9soe4jdmpheFBU6/Nsru0qyQpaG0gtdZF2wdVbi\nAFpy\r\n=ykJF\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210224.8610f7b.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210224.8610f7b","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210224.8610f7b"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210224.8610f7b_1614152844340_0.6665739433346298","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210224.dd555e8":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210224.dd555e8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210224.dd555e8","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"ace3e2cbaa3d34e5cb243967f54d9c9b3862600c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210224.dd555e8.tgz","fileCount":30,"integrity":"sha512-rREN1NdqMLpOlQlAbd5/hE2FA33+5v/Pe9KRwy32CA53hdtUtL5DlNV1gyqyvxXwycbMnNRpu3PJ/MN48tXGdA==","signatures":[{"sig":"MEUCIQD4t7TJ8eUsVhdI0fenIYgt6F12a29gMgl8+/ch2ZMcigIgLfPIkTg2zhP254nNepG/P5Ka1/3hjbp6OrKf8PL/IlQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126561,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgNuEJCRA9TVsSAnZWagAAw74QAJCqLYU4t+JR22RQ9Mqp\ngEdtCnWxf1T8EI0FIYrbNR2pxD9LHLV1YOj2gS06p92K+U2lpsZ39WsJenhG\nO5yVyVjI81pTRmaVkIVAszbrH6zjRF4dGm59ZkudHfYlLmmjKZ4EOETifvK4\nzbEOzNHCI7yV72sMwpE8B6s4Kfk6w4Ydf4hvLILsnFlLPFpaPdW97jJPXOIp\naRNvpBmJngbI9kh8mL/PuzEEhKtXcTuxb6m6NVepIkFfkDQddz3i5PxvBQsk\nTpJuzBhpd2T2LeAsJpYqnAEH1UwdBDRTUz+cl3RYJSuSSCPPdYdrQejRPj5E\nkcrqpeNgR+NPe5K+mk5xyS0ipx1ZD3lNxXo9/GtfxecokNpE7sOtOmqUxl5r\nuRl6quzTiARBuTlT0aQDPyD5XtSvGs0N491axTYF5Qq+m4hAzYxtVDE9FC5f\nQAP+WrkRW00HnWRHVqXKRDmrkXAxTU94XtydKCeDwVsu9LNo9IrXkI6seKZQ\nJcie4IDNHEwaI111dZnbT1P9NEQNx7Ih6tWvOt8lk4I17TISJmYR66iMMLMw\niEoBpGb6+F/czolMyFKiKzWCXVdwJSKKc5Yq7JZBYiM3WZaEPZXMgqU9Nmjl\nQMNHYLPCrJe8V3iJ1tulw4XKgYZdMXSEn8rJpkXcznsBCfSGesR8iiwbk3/O\n5MP/\r\n=hfp4\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210224.dd555e8.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.10","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.4","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210224.dd555e8","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210224.dd555e8"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210224.dd555e8_1614209289038_0.03537843935483309","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210225.6744890":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210225.6744890","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210225.6744890","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"630cf170e591e229f2ec7f553d4a36d479b79b9c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210225.6744890.tgz","fileCount":30,"integrity":"sha512-VH5ArKj/bUSWgFTH3G236euWX/lsQohHkjHoYbHVlyNkAiuZ5Q6sQ3MVAuKuco4LRQeMZZ9XMUIJFbSGgbQSAg==","signatures":[{"sig":"MEYCIQD28y4yTXG9KDmHjkat1A+Cgoj8evWhbClFVt0sHjuU5QIhAPR2YrMG8rJlHMSQuDhHXdlGuOYOvu46pVUL5PyMqsgm","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126561,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgN008CRA9TVsSAnZWagAAhj8P+wdhp6045Qx80qhS3Okc\n75P+9xKnHSyQ0wK61E/kBKqFlHwrWYFxRuHQC2vOSqWtzzKqHSdqDJk9mQif\npJ2g07CDv9ASzNVvqDyt68Rt5ynpk5PJfMbQBvVoiuNfrcuXBJH+4XM6ZAlx\nsLzf6L9Rsd6GjSp677Jghk+KtF/NSstFwPcXPi5v2LDlo7+eOrHdkmbeNlat\nwgxf69HdDek+rtOZQ4q2JPD+c4mvTHvnQ6yTvXFGjOMLSbZBMmjFCIRzXV/0\nWaf4QywNi9XhALur0Dau4DMjg6s8osAZiCYGquglM/OgtV+ijKEpmDo/HoKD\nayCKxtVw5l1Ddouc0jQMqnlAjO5X9vyK7eM10WSkE6yLO/IbOcMJyjTeJ3aE\nlJ55YkcLRP/UcskOyAHvyAzhzIHS04QsFS+RnCrq1A8hIFlnOqRg3y7ibC25\nqogBbTRkIjeNGERLpFFhFSo5xe29h3nY8w1e6szH+nTlPBLQ7nPmN71etiz2\nWJX3K+IyoICKfZG4jh4gOkFaes0acEKmFN2wsb7R7UdpAxBASfv9pcaWRoVD\noXn64ZNZfUgkzvGrSSjZRZbl3lGDalIxHsFaU36YTT8IaiXDSC0dZvuBZF2T\n65jMaUFTflGSRMN/NzCF5iUnh4tSzySVnHGdrwQhZrZsT72LD8Zo488fdvjf\n+941\r\n=4CDJ\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210225.6744890.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210225.6744890","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210225.6744890"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210225.6744890_1614236987606_0.7092996563498375","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210227.ba15791":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210227.ba15791","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210227.ba15791","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"928628460290cbad9ad24456340023a7b96ce6f8","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210227.ba15791.tgz","fileCount":30,"integrity":"sha512-pc4mklOSz29LIz4HG8VGFCb7mq85SDVwQrOP1OsYOAUTGBfaBazIMSSFnWGphrQMx+9XBrSanYFtqxeJ7mxDlg==","signatures":[{"sig":"MEUCIAuLuQn9JOZ9wliDqWXjujtJrfEBP39U4L/nYtXDYAELAiEA7SgTd1XsZ5TZnQWdNr8VYoUHNISTfWEUfggRrcYnV4U=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":126553,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgOfAWCRA9TVsSAnZWagAAUOgP/RJCjcrqVxAGmTPz+suD\n90l+LSz7KYCyI8M+APFYzz5kUaEte5yDyyjRAZDyN+i6+nLnnADaGIg4FVVR\nF1SsVTpd6hxPBMhEgOrlsjrxj6Zq+hSBGAnx4G9VcJKnVsbPW5NWF15EKDqZ\nwJMte5DbuNKnGC2VujILEiJh8aCzUFaBIljXd/k3cDrBstEyjndyYI42JGeZ\nYyJ4X4RO8NXct8R08kHahMjAMK0OQfafLWe9r88OTW8rKv3Lm9w8zIO64TWH\nZIaebuu3RTXdRSdK1CzyeVIOr07PIOIBCOmEHpArGDGHfPxPfyEkMVEpUDum\nxliyHfcyWZZHN6bS855EUsRrG9ZRF7q/CBw55C8nfpgeEFGbJ24MoQNOyGvJ\nusPtcwrJarkrc39xcCD8JoduEZFqXbjyWcTTbVXEFzLoP/kalUQAJiO4GNIt\nGfyBrIHGEBwoWQBn3t3iwhTfgouyU/9yaWjTy7UofKyL1ov7ihkjOpf/cJv8\nPdn+NbakxdF31APR/E+bsR0vrgZiY31mJWpbH152BfmzQRksRaF+BFX6C8hr\nT3SXaZsuJ6odjoCpVKKq4udhQASoAA9T1iHqTVj8G+FSrXkIrNjguGNQ8Bva\nAwpnCvs2sI78CxR3tk4iY84Q2ZJXLrU5nX4fWIzMHMRoAmbgD+RyAqHiFyu4\nJde/\r\n=iYhR\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210227.ba15791.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210227.ba15791","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210227.ba15791"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210227.ba15791_1614409750434_0.9934462391116368","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210310.a728f41":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210310.a728f41","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210310.a728f41","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"3d78bbf85aca868dc4bbc03d1f8cc08605632390","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210310.a728f41.tgz","fileCount":30,"integrity":"sha512-NeYDOp4YDqEdCshhJRfj39FLidTQ99mffTcy8ymCka4fgHk3+F2TkWCHmm7Fl9FrDqf6Iv0Kq7qJ76Uxyr0LVg==","signatures":[{"sig":"MEUCIGoWt4G58lVXK3CcZfVq/CcUdYt6JyE7kiLZWu2qfvTfAiEA2wfV2pLfcBrR3PPNYeBr0QT4VE1TtYelVX6DmpCj4zg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":125824,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSQTGCRA9TVsSAnZWagAAI+0P/RRsWx9zxowCQpp5TkGK\n/9z7jy1kRMTp8w8oN+XCH09zVOFSxlfRSemnXu4NvJ2hWJVFiWp2MpIVC3bD\nAor13yw8UGQfEgwipvYAEMJljBNCUrJ1OOa7wKmcVTX3uAw/C1MoiV7wTJ4T\nXRfRmOsAq13FvRa9PRfeecOTjponjawdPDzsZVe7W7/r9E6mV152WH3Wx8vF\nGuaaMpyqTdnBw5cN1hapI0cUzAY4zocDMjNKcLNVEIhUiihS+Vxa6W6crQh2\npXkqnLAjsyuMz9SFnevTmcCAbrQ1+ZL8vaZ7jmSfW5Bx9OlLMuMmVWCaXITo\nQ7n4F6VaWB91csZqWmG7Mxme2OF4jHVvdfuEPnc82r97FC/fxQn1wBDAMp0y\nAufNuthm/tBEJBOLrKRcZuUlK0aTfhMuP9ra1QEPHnSRXPP+eo/eZ6MOsjrk\nYLKreyeVCweRdpMqZ0/5YhsRsb9f7eQ8pEMiRy88TTkDUm+YSOPPHUEALqww\nIuoye4/vg4iGfP8spgsI4/R3/iTxgWuyCdP9B5TsVzucafTB71bnVH4F/dVt\nwTN6Gs0if68/kBUUAI3OniPYcKY0OJ506CNWz3Wh0wmmgkQFb9LjhwgBiUIg\nmFC2KsSQBdRaMl7TPznBapxNhGwDaI7bAchbubLAtiWnEANdyQEEEZHLNarG\nzal3\r\n=dSbE\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210310.a728f41.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210310.a728f41","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210310.a728f41"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210310.a728f41_1615398085535_0.19244023206669714","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210311.6018ddf":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210311.6018ddf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210311.6018ddf","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1332edb6863857ff322f6ad4a6b82841797a6624","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210311.6018ddf.tgz","fileCount":30,"integrity":"sha512-ulLL6kFeacTfqxjdsYQKImY0XKJwmmWkrFuHpzC/ykUEVIxJwWOk4OLAG80u6AAprJMWLJoCdn9ml3D7bPVHFw==","signatures":[{"sig":"MEUCIQCQtAqPhHDVPNZCd0LqSkn7dPwQoE0w1TKOTbrYUoMbcwIgInIXf0FDTq9/cP7uatjG1zI/QmdMx8cIhzjvNp7qdLU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":125824,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSlSeCRA9TVsSAnZWagAAz3cP/3gVHMwlD4opON38Da8n\nrNZsKwikH5onEa9A9FEA9L7oQTR360s0ObTLpGLyJwHHDbATMwnlRHCRFIAq\nBk5obEa0Ae+tJBSd0OO7sOiSdkT8qV6jdbcojNPLjJBtfV2TNn+BV2OOhO03\nLmg+oA9D0cN1fuVKoN4Ex0l6hEvw2AAo67TIX9ZAhnYgS7zZpNvkrsuDXVrH\nvPT4opqai5NLbz8XFvNFdSpbA/hlVoN8Nz3Anzr+6fV1/o+gVrTPgAkjHCbR\n0U2Aj44hy+GZJcWujMn00nGs7zCN20ThDaC/CsPchPtg5ifG4qKPkAySdEqK\nHe+/nVIvobOrS6x9QwIkddTnZqoQN52HkCbUd14zSd8b31ss1MzJ8FEqVIl+\nVzk9EwkkCqhVjv6FDkZlivkyXPHqq2EoIJsWS+lP+GUflunzup6Ln8eada5P\n43HbIqDFQ+wuWm4pE26WStEls+6sn97lSqnYtucbIBlVr4GuQSlNv//H9b+M\nLjF7i1bFs2Uf9CpQjSr3jUDWb/9RUgHX429cKwaN35lXdVSxCaEy4orlV24W\nIGW4eCd9QX/gkRcdObkgTbA5QYMZiplhifpyJRXX4uoNDSaVnBhDWBrG202t\nYKKwRnvVfTiv4VF9m2nAWxqm9z4lkOnDZnVqAok6gov9fe5UMk/TnjPEOsoM\nyJYJ\r\n=lou0\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210311.6018ddf.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210311.6018ddf","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210311.6018ddf"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210311.6018ddf_1615484061667_0.31512272704060806","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210311.98dc244":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210311.98dc244","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210311.98dc244","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"67aaab40a6b04683960e299cc841c2a09db66448","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210311.98dc244.tgz","fileCount":30,"integrity":"sha512-qn/p2hARGMmaexV2qpTas4fvEysDQYuQd6XD50Fvvwyr4bkOQe7QFb2QyuLNjJ7O4d8wusbLpVCXy33X94dDkw==","signatures":[{"sig":"MEQCIGCc1aDg4zK+nATF91LbDaKr2Chm6VH5QMGAfE3rekuVAiB7kh5KRikNG6SzIg5kLQCYg15DYp9+yliWLYvcwSNwtw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":128890,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgSnH9CRA9TVsSAnZWagAA8A8P/RtB393UUI/ax7mblUII\nnNXEN8gaDrAKwOhsjaGWCUWoov+QpEs73ZFHBfLedx2xKnzQ4BanIn0LDg0/\nQGr3Up3YlUyc712/aZOmAYBDw6x1kOGo5iBqrmarfrIOtwax3dabTT408F+K\n4rJ7pXMvt0UdAHL0IsYDb8lcC5+vlWr18grfPpM2dFZ9l+WAsDEbpmcYErwH\nAd8aqAx3V1uRHtC3CCFNiKgTEh+ZN+ESNevnlVuXNIpreriJImutqvnVVIcE\n8JATvT9wq15cBW5utOKoyNCyLOLjrpZDg2QEh04OiT3SF715Z6/iX80RVj4G\ncqY03DNFeghuaRVDN2x1GZG9RtbEuxvBpQMnkgXwAxhZ/aFDlU6D0nPD/P0w\nM8jA/XZH+7/uOg9ixwl6ggr0oaqED4JG7bWICWZePQU1sHxlROCOiPcDIHpf\nvnPRsmpGJmfTtJUAXHIbBcwU/+7ruosj7RAF/ocFFU5lmy6mGu8x1WvQboTl\nGvv8OlH4VLbGdzBGPeJyFM3KPRHzzXn006P3zOAg/OmpzdQb1MXxwuedH2jQ\n1SsoH8h4w2RpYlaz1W0CSpF+aub0Tiiq7jUOmfAcrs7TPCRN0KRP8WjtVg8e\nNCzUKpz53fGgV4ZQwHWpcUBaX3NrRzu6vaHiwhIz5nbLwkr8N4ZbNYigYYJo\nUde7\r\n=Wc4t\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210311.98dc244.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210311.98dc244","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210311.98dc244"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210311.98dc244_1615491580730_0.8343353739782426","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210316.cdd0819":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210316.cdd0819","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210316.cdd0819","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1f03dce8b7a0e59a4619d4fedd83905cd6674259","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210316.cdd0819.tgz","fileCount":30,"integrity":"sha512-2Qum5lOPDSLSW5VCVX96g+1xYURzgTRUwgdvzoW73NbZfnjHCKlv/aQGc4fVWe2DJizll+OyB5XPfAKFUA4cDw==","signatures":[{"sig":"MEYCIQC9aqmUqd1vfn6PN9lrYhxY7rGQzI55HM2XNGKh4XgzbAIhAP9IY/yarjA1iioFAIHtPHWGCf0ulmqOFsjMyEhf+jqg","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":125824,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgUR40CRA9TVsSAnZWagAAMyYQAJeNgKs9K3bKbu33xLOj\nkuEDi2wXHzEOGSqosa64iIhYKh2Rxrf+PbAD9HCHxKr+nkjGCtTTiw/AW4Cy\nMrecTGU+QwNPR4ivGGZmqA5DX2YOtIY6SMUqP4DmutBoFXggZK5zQnF2VZ9J\nDkvTdS7fsLcST4ekop/j6/O3DooLAs/kn9d36BRVeJwah4e04g0OkLvfClUv\nyB24paHDaihcHra1Dnw4FFM6YUxEsWQTapGxuHirXuIjwddX1iPvQAgcWXvr\nDFdNGo1NBaFmQ/EVqe82rz/uWh/n4eHQSLThA5N/gTsukS1ui3FuU/5YBFKg\nv8Apy+PONkq9pbMg1cbIvhrsr1XfPmRqWrqFHt9eKvS72KZRTf+tPHdMT0np\nC5nGoJbM0Mr+QT7FbePJDycMlqJFYfDzknJ5Ek9zv1Lnc31zbcuSrCGFfceH\ncKNcIb88s6agrcIZKj8HyV+Uz8q+En8s1z3PElihwFHlHYtKPZ3cvFNUk+iz\nMkit4YR2wH3Ix+q/R9ExgtdlEBCljypOroj7yxBGofjYqfjXlSHVEsSjVOHs\nUaddMtJolRk2pbqsEJcN+PPhcQqaDvxFclQQiZUrlNBj8bnOsBkzBZqMll+Q\neqVTFit9n/u0xgjwIGRsAuOZUKRTDqgJ6WVPpVfof3RfACCUb2Ib+0Tc+gza\npNwZ\r\n=Iz46\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210316.cdd0819.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210316.cdd0819","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.12.0-beta.20210316.cdd0819"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210316.cdd0819_1615928884043_0.034357051706081165","host":"s3://npm-registry-packages"}},"4.13.0-beta.20210316.e8ec340":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.0-beta.20210316.e8ec340","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.0-beta.20210316.e8ec340","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"48bbdd0271b8ebf440cc232156ae8c9eff7fcbe9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.0-beta.20210316.e8ec340.tgz","fileCount":33,"integrity":"sha512-1gzmPEFWEDaoupCvDHM+99s0OEPyC3Jru4UDwkLha+LKuoRHvG2uHMxrMxJGGRkeCuO99uHm6u/NbTaz8aKnfw==","signatures":[{"sig":"MEUCIB0dXZ8FzvqIIqWzMWopwBcj5Jmvs8QJBvZbK95UoPhQAiEAgu/qqAlp7twpSAACWlNtuB8HX7h/ZDveoiPJmH9deN4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":137133,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgUS60CRA9TVsSAnZWagAASQoP/AqtObukGoMkvL0oWG4I\nu7p+DPCIk187uiOngyHlvdkcaYBjfJrSKQypkXNmZijeaQS8/oaKigz2EO9e\nlGDq+f9yfdHPMYJI8QmcuvW+lTk+XbCruSLlM+hPSWvvAMrdFxOzDSocHfgn\nl35PJxBtV4PrM/d7Y+q7Vr2Lz2io5tMeIFdsdwGKB1hLKQ8G7GlRABBFVxrw\nitCqHOaSsWHAtV82BpUi7zciiZXufk3Oy39t4mC9/dYjMp2ekrSzSbp9ZV6W\nAmKZ2u8nBGWw6UkMNW01/y0cKPUMVK8hHQT6gNeitXEXQ2q00zfW+WsL0mgt\n3mfeaFb0zQbTxomIdFqxuBnChlQvI1x6Ism+Zxbm02CrTr63F4mOgcC1Qy8g\nKRHc2haCw4bLRR3xv5fmJFQShw9GkAn9cPDmZEp7/IuKoDhliAy7x6D+VrDt\n2mQoAnD/ddKHIYrId/i+hsv5+O2QmIsPebmpFTOgmL/ohQpb+SYm81UwxO86\npPxlxKpmOPXZADkBKiIub2MeDsQrIiByLklW42fAn3sxDxsaPtRRViEFQjTq\nyOKYbWc3tdGU1XbJoksStturJQBuVvKaManyVMgLgu5Zl7ZRv4IqMYhgJctj\nvMtcC/UTQr2SEQhxx7W4+cg3/3AmF5kTMnFHJn+9B+V8TMb8LoBO7If+QhB0\nq3N5\r\n=JiUZ\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.13.0-beta.20210316.e8ec340.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is a plugin for @microsoft/botframework-cli.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-luis-cli.svg)](https://npmjs.org/package/@microsoft/bf-luis-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n## Prerequisite\r\n- Supported OS: MacOS v10.14/v10.15, Ubuntu 18/20 or Windows 10\r\n- [Node.js](https://nodejs.org/) version 10 LTS, 12 LTS or 14 LTS\r\n- @microsoft/botframework-cli\r\n```\r\n$ npm install -g @microsoft/botframework-cli\r\n```\r\n- For installation on Linux Ubuntu LTS, run the following after bf plugin install (see installation below):\r\n```\r\nsudo apt-get -y install libicu-dev\r\ncd <$(npm root -g)/@microsoft/botframework-cli/node_modules/orchestrator-core>\r\nln -s libicudata.so /usr/lib/x86_64-linux-gnu/libicudata.so.60 \r\nln -s libicuuc.so /usr/lib/x86_64-linux-gnu/libicuuc.so.60\r\nln -s libicui18n.so /usr/lib/x86_64-linux-gnu/libicu18n.so.60\r\ncurl -LO -C - https://github.com/microsoft/onnxruntime/releases/download/v1.2.0/onnxruntime-linux-x64-1.2.0.tgz\r\ntar xvzf onnxruntime-linux-x64-1.2.0.tgz\r\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd):$(pwd)/onnxruntime-linux-x64-1.2.0/lib\r\n```\r\n- For installation on Mac:\r\n```\r\nbrew install icu4c\r\n```\r\n\r\n## Installation\r\nTo install the bf-orchestrator-cli plugin:\r\n\r\n```\r\nbf plugins:install @microsoft/bf-orchestrator-cli@beta\r\n```\r\n\r\nTo uninstall the bf-orchestrator-cli plugin, which is a necessary step for upgrading the plugin:\r\n\r\n```\r\nbf plugins:uninstall @microsoft/bf-orchestrator-cli\r\n```\r\n\r\n# Commands\r\n\r\n\r\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\index.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator create command help\r\n -i, --in=in The path to source label files from where orchestrator example file will\r\n be created from. Default to the current working directory.\r\n Valid file extensions are lu, .qna, .json and .tsv.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Path where generated orchestrator example file will be placed.\r\n Default to current working directory.\r\n --hierarchical Add hierarchical labels based on lu/qna file name.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:create \r\n $ bf orchestrator:create --in ./path/to/file/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/\r\n $ bf orchestrator:create --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -o, --out Optional. Path to where Orchestrator base model will be saved to. Default to current working directory.\r\n --versionId Optional. Base model version to download -- reference basemodel:list output for options. If not specified, default model will be downloaded.\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -r, --raw Optional. Raw output\r\n -h, --help Show CLI help\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nA user can play with an Orchestrator base model interactively and improve a snapshot in real-time.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive --out= --model=\r\n [--entityModel=]\r\n [--in=]\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'interactive' command help.\r\n -i, --in=in Optional path to a previously created Orchestrator .blu file.\r\n This argument is optional users can use the 'interactive' command\r\n to start an Orchestrator snapshot from scratch. The 'n' commandlet\r\n can save the utterance labels into a snapshot (.blu) file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n\r\nDESCRIPTION\r\n\r\n The 'interactive' command is an interactive session that a user can access an Orchestrator model in real-time\r\n doing following:\r\n 1) Predict the intent of an input utterance using the 'p' commandlet.\r\n 2) Analyze a model example set, by executing the 'v' (validation) commandlet and produce an evaluation\r\n report in real-time.\r\n 3) Add, remove, or change the intents of an input utterace using the 'a', 'r', and 'c' commandlets,\r\n respectively. Users can reference a validation report and choose an ambiguous, misclassified, or\r\n low-confidence utterance and change their intent labels.\r\n 4) Remove some labels completely from the model example set using the 'rl' commandlet.\r\n 5) Create a new model example set snapshot using the 'n' commandlet.\r\n\r\n Below is a list of the commandlets that can be issued during a 'interactive' session.\r\n\r\n Commandlets: h, q, d, s, u, cu, i, ci, ni, cni, q, p, v,\r\n vd, va, vm, vl, vat, vlt, vmt, vut, vo, a, r, c, rl, n\r\n h - print this help message\r\n q - quit\r\n d - display utterance, intent label array inputs, Orchestrator config,\r\n and the label-index map\r\n s - show label-utterance statistics of the model examples\r\n u - enter a new utterance and save it as the \"current\" utterance input\r\n cu - clear the \"current\" utterance input\r\n i - enter an intent and add it to the \"current\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n ci - clear the \"current\" intent label array input\r\n ni - enter an intent and add it to the \"new\" intent label array input\r\n (can be an index for retrieving a label from the label-index map)\r\n cni - clear the \"new\" intent label array input\r\n f - find the \"current\" utterance if it is already in the model example set\r\n p - make a prediction on the \"current\" utterance input\r\n v - validate the model and save analyses (validation report) to\r\n \"experiment_predicting_va\\orchestrator_predicting_set_summary.html\"\r\n vd - reference a validation Duplicates report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n va - reference a validation Ambiguous report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vm - reference a validation Misclassified report and enter an index\r\n (previously generated by the \"v\" command)\r\n for retrieving utterance/intents into \"current\"\r\n vl - reference a validation LowConfidence report\r\n (previously generated by the \"v\" command) and enter an index\r\n for retrieving utterance/intents into \"current\"\r\n vat - enter a new validation-report ambiguous closeness threshold\r\n vlt - enter a new validation-report low-confidence threshold\r\n vmt - enter a new multi-label threshold\r\n vut - enter a new unknown-label threshold\r\n vo - enter a boolean for obfuscating labels/utterances or not in evaluation reports\r\n generated by the \"v\" command'\r\n a - add the \"current\" utterance and intent labels to the model example set\r\n r - remove the \"current\" utterance and intent labels from the model example set\r\n c - remove the \"current\" utterance's intent labels and then\r\n add it with the \"new\" intent labels to the model example set\r\n rl - remove the \"current\" intent labels from the model example set\r\n n - create a new snapshot of model examples and save it to\r\n \"experiment_predicting_va\\orchestrator_predicting_training_set.blu\"\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --out=resources\\data\\Columnar\\InteractiveOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query --model= --query=\r\n [--entityModel=]\r\n [--in=]\r\n [--limit=]\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator query command help\r\n -i, --in=in Path to previously created Orchestrator snapshot (.blu file).\r\n -q, --query=query Query string to predict.\r\n -m, --model=model Path to Orchestrator base model directory.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -l, --limit=# (optional) Limit of number of predictions.\r\n\r\nEXAMPLE\r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict \r\n $ bf orchestrator:query --in ./path/to/blufile/ --query /query/string/to/predict --model ./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/query.ts)_\r\n\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\n Please see below for detailed help messages on arguments and requirements for the three modes.\r\n Notice that the input specified by the '--in' parameter may be different from mode to mode.\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- test mode`\r\n\r\nTest utterance/label samples from an input file and create an evaluation report.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -h, --help Orchestrator 'test' command help.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -m, --model=model Directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -t, --test=test Path to a test file, or comma-separated paths to\r\n a collection of test files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The 'test' mode can test an Orchestrator model and example snapshot set against a test utterance/intent file.\r\n It will generate evaluation reports and auxiliary files.\r\n This mode is activated if a \"--test\" argument is provided to the \"test\" command.\r\n Please refer to the 'evaluation' mode for details of the evaluation report.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\TestOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu --test=resources\\data\\Columnar\\EmailTest.txt\r\n\r\n Notice that the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies a downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- evaluation mode`\r\n\r\nCreate leave-one-out cross validation (LOOCV) evaluation reports and axuiliary files on\r\na previously generated Orchestrator .blu snapshot file.\r\nThis mode is activated if there is no \"--test\" or \"--prediction\" arguments provided to the \"test\" commands.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a previously created Orchestrator .blu file.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -m, --model=model Optional directory or a config file hosting Orchestrator base model files.\r\n -e, --entityModel=entity-model Path to Orchestrator entity base model directory.\r\n\r\nDESCRIPTION\r\n\r\n The 'evaluation' mode executes a leave-one-out-cross-validation (LOOCV) analysis\r\n on a model and its example snapshot set. It also generates a detailed evaluation report\r\n with the following sections:\r\n\r\n > Intent/utterance Statistics - label and utterance statistics and distributions.\r\n > Duplicates - tables of utterance with multiple labels and exact utterance/label duplicates.\r\n > Ambiguous - ambiguous predictions that there are some other label predictions whose\r\n scores are close to the correctly predicted labels. Ambiguity closeness is controlled by the \"ambiguous\" parameter, default to 0.2. I.e., if there is a prediction whose score is within 80% of\r\n the correctly predicted label score, then the prediction itself is considered \"ambiguous.\"\r\n > Misclassified - utterance's label labels were not scored the highest.\r\n > Low Confidence - utterance label labels are scored the highest, but they are lower than a threshold.\r\n This threshold can be configured through the \"lowConfidenceScoreThreshold\" parameter, the default is 0.5.\r\n > Metrics - test confisuon matrix metrics. Please reference the \"assess\" command description for details.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\Columnar\\EvaluationOutput --model=resources\\data\\Columnar\\ModelConfig --in=resources\\data\\Columnar\\Email.blu\r\n\r\n Notice that it is not required to load a model file if the input is already a snapshop .blu file.\r\n Nevertheless the ModelConfig folder is created by the 'bf orchestrator:basemodel:get' command.\r\n Inside the \".../ModelConfig\" directory, there is a \"config.json\" that specifies downloaded base model files\r\n among other hyper parameters. Here is an example: \r\n {\r\n \"VocabFile\": \"vocab.txt\",\r\n \"ModelFile\": \"model.onnx\",\r\n \"Name\": \"pretrained.20200924.microsoft.dte.00.03.en.onnx\",\r\n \"Framework\": \"onnx\",\r\n \"Publisher\": \"Microsoft\",\r\n \"ModelType\": \"dte_bert\",\r\n \"Layers\": 3,\r\n \"EmbedderVersion\": 1,\r\n \"MinRequiredCoreVersion\": \"1.0.0\"\r\n }\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n# `bf orchestrator:test -- assessment mode`\r\n\r\nCreate an evaluation report on assessing prediction against ground-truth instances.\r\nThis command can execute an independent assessment of a prediction set against a ground-truth set,\r\nit does not require a base model that other Orchestrator commands may need.\r\nThis mode is activated if a \"--prediction\" argument is provided to the \"test\" command.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug Print detailed debugging information during execution.\r\n -i, --in=in Path to a ground-truth label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n -o, --out=out Directory where analysis and output files will be placed.\r\n -p, --prediction=prediction Path to a prediction label file, or comma-separated paths to\r\n a collection of files -- most uselful for crosss-valiaton.\r\n\r\nDESCRIPTION\r\n\r\n The \"assessment\" mode compares a prediction file against a ground-truth file, then\r\n generates two detailed evaluation reports, one for intent prediction and the other entity, along with some\r\n auxiliary files.\r\n The input files can be in LU, LUIS, QnA Maker, TSV, or a special JSON label array format (described below).\r\n\r\n This command intends to provide a comprehensive and consistent multi-class model evaluation tool in\r\n order to avoid teams reporting their own inconsistent evaluation metrics, especially in a contest\r\n evaluating competitive technology or models. Below are some sources of evaluation inconsistencies if\r\n evaluation were conducted and metrics reported by the teams themselves:\r\n\r\n 1) Metric - one team may report Micro-Average, Macro Average, or else. As shown below,\r\n there can be many ways in computing a metric average, and\r\n the discrepancy can be huge using different averaging formulations.\r\n Sometimes the definition of a metric, say macro average, can even vary among different teams.\r\n 2) Datasets – even though every party tests on the same dataset source, the dataset might diverge\r\n after a while after some data massaging, sometimes due to processing errors or filtering logic.\r\n Some teams may prefer some datasets, but not others, for their particular technology or models.\r\n 3) Data processing – some test dataset can contain duplicate utterances and labels.\r\n De-duplicate or not can affect final metric calculation. \r\n 4) Confusion matrix formulation – what is a TP, FP, TN, or FN? \r\n Even though their textbook definition is clear, they can still be up to interpretation\r\n in real world scenarios. For example, TN may not make sense in evaluating entity extracting.\r\n 5) Label interpretation – Sometimes people might ignore some test results if their prediction\r\n scores are too low. We should have a consistent way to include those predictions. On entity, some\r\n people may choose to evaluate based on per-token tags, instead of per-entuty-mention. \r\n 6) Label processing – what is an UNKNOWN label? Different teams may have their unique strategies\r\n in processing empty, UNKNOWN, or never-seen-before labels in the ground-truth and prediction files.\r\n 7) Evaluation Tool – every team has their own tools that might not be consistent in metric computation.\r\n\r\n The \"assessment\" mode aims to address these issues:\r\n\r\n 1) Metric -- the \"assess\" command calculates many average/aggregate metrics to satisfy a variety of evaluation\r\n needs and metric definitions.\r\n Which metric to focus on for decision-making is up to an evaluation committee or stakeholders.\r\n The tool has no bias, even though each party might have its favorites for its own agenda.\r\n 2) Datasets -- it's necessary to create a dataset repo for the community to share\r\n ground-truth datasets and predictions. Again it’s up to an evaluation committee or individual party to choose the\r\n evaluation datasets (and performance metrics) compatible with their projects and scenarios.\r\n 3) Data processing – this BF-orchestrator evaluation package provides consistent logic in processing datasets.\r\n For example, it does de-duplication, so an utterance and its label won’t contribute more than once\r\n in metric calculation.\r\n 4) Confusion matrix formulation – again, this BF evaluation package provides consistent formulation logic in\r\n intepreting the 4 confusion matrix cells.\r\n For example, entity evaluation does not have TN as the combinations of entity offset and length not in\r\n the ground-truth or predicted set can be limitless.\r\n 5) Label interpretation – again, consistent is key and this \"assess\" command does not silently\r\n ignore some test results for whatever reasons. Every test instance should contribute to\r\n metric calculation unless they are spurious (test intent/entity name not in the training set)\r\n – due to processing mistakes by whoever prepared it. \r\n 6) Label processing – this BF-Orchestrator package pre-processes labels and treats\r\n an utterance’s empty, None, and never-seen-before labels as UNKNOWN. Since BF-Orchestrator allows\r\n multi-label intents for utterances, UNKNOWN is stripped if it con-exists with known labels\r\n for an utterance.\r\n 7) Evaluation Tool – BF-Orchestrator has been released to the world (through NPM), everyone\r\n in the world can run it, file a bug or PR to improve it.\r\n It’s not a script hidden in someone's laptop or some team repo.\r\n\r\n Besides addressing these evaluation inconsistency issues, the \"assess\" command also produces a\r\n comprehensive metric report as well as label and utterance distributions.\r\n\r\nINPUT\r\n\r\n The input ground-truth and predictions files can be in LU, LUIS, QnA Maker, TXT, TSV or a special JSON array format.\r\n The TXT file format only supports intent labels and it must have 2 columns, 'labels' and 'utterance',\r\n sepatated by a TAB. The 'labels' column can contains multiple labels delimited by camma.\r\n\r\n For entitiy labels, a user can choose LU, LUIS, or a JSON array format\r\n that each entry contains a labeled utterance following the schema and example shown below.\r\n In the array, each JSON entry has a \"text\" attribute for an utterance. The utterance can have an array pf\r\n \"intents.\" The utterance can also has an array of \"entities.\"\r\n Each entity entry contains an \"entity\" attribute for its name, a \"startPos\" attribute indicating the offset\r\n of the entry in the utterance, and a \"endPos\" attribute for the final location of the entity\r\n in the utterance. The entity can have some optional fields, such as a \"text\" attribute\r\n for a entity mention within the utterance, however this attribute is only for debugging purpose and not consumed\r\n for the \"assess\" command.\r\n\r\n [\r\n {\r\n \"text\": \"I want to see Medal for the General\",\r\n \"intents\": [\r\n \"None\"\r\n ],\r\n \"entities\": [\r\n {\r\n \"entity\": \"movie_name\",\r\n \"startPos\": 14,\r\n \"endPos\": 34,\r\n \"text\": \"Medal for the General\"\r\n }\r\n ]\r\n },\r\n ...\r\n ]\r\n\r\nREPORT\r\n\r\n The \"assess\" command reads the input ground-truth and prediction JSON files and generates\r\n distributions of the utterances and their labels. It also groups each utterance's\r\n ground-truth and prediction intent and entity labels together, compares them, and determines if\r\n a prediction is in the ground-truth or vice versa. Using these analyese, the \"assess\" command\r\n generates HTML reports and some auxiliary files at the end.\r\n One report is for intent evaluation and the other for entity.\r\n\r\n Each report has the following sections:\r\n\r\n > Ground-Truth Label/Utterance Statistics - ground-truth label and utterance statistics and distributions.\r\n > Ground-Truth Duplicates - tables of ground-truth utterances with multiple labels and exact utterance/label duplicates.\r\n > Prediction Label/Utterance Statistics - prediction label and utterance statistics and distributions.\r\n > Prediction Duplicates - tables of prediction utterances with multiple labels and exact utterance/label duplicates.\r\n > Misclassified - utterances with false-positive and false-negative predictions.\r\n > Metrics - confisuon matrix metrics.\r\n\r\n In the Metrics section, the Orchestrator \"assess\" command first generates a series of per-label\r\n binary confusion matrices.\r\n Each binary confusion matrix is comprised of 4 cells: true positive (TP), false positive (FP), true negative (TN),\r\n and false negative (FN). Using these 4 cells, Orchestrator can compute several other confusion\r\n matrix metrics, including precision, recall, F1, accuracy, and support, etc.\r\n Please reference https://en.wikipedia.org/wiki/Confusion_matrix for details.\r\n\r\n Notice that the logic constructing a per-label confusion matrix is a little diffenent between intent\r\n and entity labels. Here are the logic outlines:\r\n\r\n Intent - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth intent array and prediction intent array. If a label is in\r\n both arrays, then this utterance is a TP for the label's confusion matrix. If a label only exists in\r\n the prediction array, then it's a FP for the predicted label's confusion matrix. Similarly\r\n if a label only exists in the ground-truth array, then it's a FN for the ground-truth label's confusion\r\n matrix. For any other labels, the utterance is a TN for their confusion matrices.\r\n\r\n Entity - the \"assess\" command iterates through one ground-truth utterance at a time\r\n and compare its ground-truth entity array and prediction entity array.\r\n If an entity mention exists in both array, then it's a TP for the entity label's confusion matrix.\r\n An entity mention is comprised of the entity name (label), start position in the utterance, and the length\r\n of the entity mention. An entity mention is considered a match to another only if these three components\r\n are exactly the same.\r\n FP and FN logic is similar to those for intent. However, there is no TN for evaluating entity mentions\r\n as there can be too many possible entity mention candidates for an utterance when consider the start-position\r\n and entity length.\r\n On the other hand, number of intent labels for an utterance is usually one or a small number.\r\n It is still possible to define a custom TN logic, such as\r\n an entity label is a TN if it does not exist in an utterance's ground-truth or prediction label arrays at all.\r\n However as some important metrics such as precision, recall, and their combination (F1) do not rely on TN,\r\n so that it's really not necessary to collect TN for evaluating entity predictions.\r\n\r\n By the way, sometimes an erroneous prediction file can contain some labeled utterances not in the ground-truth\r\n file. These utterances are called spurious, and they will be listed under the \"Prediction Duplicates\" tab\r\n in an evaluation report called \"Spurious utterance and label pairs.\"\r\n\r\n Once the serial of per-label binary confusion matrices are built, there are plenty of\r\n per-label metrics, which can then be aggregated using several diverse averaging schemes.\r\n An evaluation report's \"Metrics\" tab contains several of them:\r\n\r\n 0) Micro-Average - Orchestrator is essentially a multi-class ML learner and model, so evaluation can also be expressed\r\n as a multi-class confusion matrix, besides the series of binary per-label confusion matrices\r\n mentioned above. In such a multi-class confusion matrix, every prediction is a positive and\r\n there is no negative. The diagonal cells of this multi-class confusion matrix are\r\n the TPs. The micro-average metric is the ratio of the sum of TPs over total. Total is the sum\r\n of all the supports (#positives) aggregated from the series of binary confusion matrices.\r\n 1) Micro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate an bucket array that each bucket contain the per-label binary confusion matrix metric value as well\r\n as the per-label support. From this metric-support array, the micro-first-quartile metric is\r\n the metric value from the entry at the first quartile position in the bucket array.\r\n 2) Micro-Median - similar to Micro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 3) Micro-Third-Quartile - similar to Micro-First-Quartile, but at the third-quartile position.\r\n 4) Macro-First-Quartile - use the series of binary confusion matrices, sort them by a metric (e.g., F1) value,\r\n and crate a simple array of the per-label binary confusion matrix metrics. From this metric array, the\r\n macro-first-quartile metric is the metric value at the first quartile position.\r\n 5) Macro-Median - similar to Macro-First-Quartile, but at the second-quartile position, i.e., median.\r\n 6) Macro-Third-Quartile - similar to Macro-First-Quartile, but at the third-quartile position.\r\n 7) Summation Micro-Average - the second way coalescing the series of binary confusion matrices into one is by\r\n summing up their TP, FP, TN, FN numbers. These 4 summations are then formed as one binary confusion\r\n matrix and can be used to calculate overall precision, recall, F1, and accuracy.\r\n 8) Macro-Average - another way coalescing the series of binary confusion matrices simply takes\r\n arithmetic average of every metrics individually. The denominator for computing the averages\r\n is the number of labels existed in the ground-truth set.\r\n 9) Summation Macro-Average - While macro-average takes a simple arithmetic average on every metrics.\r\n Summation macro-average only takes the average on the 4 confusion matrix cells, TP, FP, TN, and FN.\r\n Precision, recall, F1, and accuracy are then calculated by the 4 averaged cells.\r\n 10) Positive Support Macro-Average - some prediction file may not contain all the ground-truth utterances\r\n and may lack predictions for some ground-truth labels completely. The averaging denominator for this metric\r\n uses the number of prediction labels, i.e., number of the labels with a greater-than-zero support.\r\n 11) Positive Support Summation Macro-Average - this metric is similar to 3), but use 4)'s denominator.\r\n 12) Weighted Macro-Average - this metric averaging approach takes an weighted average of the series of binary\r\n per-label confusion matrices. The weights are the per-label prevalences, which are listed in the\r\n \"Ground-Truth Label/Utterance Statistics\" tab.\r\n 13) Weighted Summation Macro-Average - similar to 6), but the weighted average only applies to\r\n the 4 confusion matrix cells. Weighted TP, FP, TN, FN are then used to calculate precision, recall, F1, and\r\n accuracy.\r\n 14) Multi-Label Exact Aggregate - Orchestrator supports evaluating multi-label intents for each utterance.\r\n I.e., an utterance can belong to more than one categories (intents). For a multi-intent application, a model can\r\n actually generate spurious (too many) intent predictions that can still boost the \"per-label\" metrics\r\n described thus far. In a \"per-label\" metric, every label (an instance can have more than one label)\r\n can contribute to metric computation, therefore some instances may contribute more to metric\r\n calculation more then others.\r\n To counter this problem, Multi-Label Exact Aggreagate is a \"per-instance\" metric that it builds\r\n a binary confusion matrix directly and an instance can only contribute to metric computation once\r\n disregard the number of labels it has.\r\n An utterance is only a TP if a non-empty prediction's intent array is exactly equal to the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. Then, if an utterance's ground-truth intent array contains at least\r\n one label not in the prediction intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n 15) Multi-Label Subset Aggregate - Similar to Multi-Label Exact Aggregate, but this metric's TP logic is less\r\n restrictive.\r\n An utterance is only a TP if a prediction's intent array is a non-empty subset of the ground-truth array. If a prediction's intent array contains at least one label not in the ground-truth array,\r\n then this utterance is a FP. If an utterance's prediction intent array is empty while it's not\r\n for the ground-truth intent array, then this utterance is a FN. A TN only happens if both arrays\r\n are empty. Notice that this metric only applies to multi-intent predictions, it is not calculated for\r\n evaluating entity predictions.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --out=resources\\data\\EvaluationJsonFormat\\AssessmentOutput \\\r\n --in=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_ground_truth_instances.json \\\r\n --prediction=resources\\data\\EvaluationJsonFormat\\orchestrator_testing_set_prediction_instances.json\r\n```\r\n\r\n_See code: [src\\commands\\orchestrator\\test.ts]https://github.com/microsoft/botframework-cli/blob/beta/packages/orchestrator/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.15.5","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"next","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.13.0-beta.20210316.e8ec340","@microsoft/bf-cli-command":"next","@microsoft/bf-orchestrator":"4.13.0-beta.20210316.e8ec340"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.0-beta.20210316.e8ec340_1615933107895_0.8755240118279191","host":"s3://npm-registry-packages"}},"4.12.0-beta.20210322.314475a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.12.0-beta.20210322.314475a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.12.0-beta.20210322.314475a","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"620c6029c610bbeccb3e9c4856fc71e08cc8b6f9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.12.0-beta.20210322.314475a.tgz","fileCount":30,"integrity":"sha512-T1kAAXfzdYowcAv0rxbR/A6XsXQzAf6qnIMz1dHJ2eb6C83BjUFOBHRJnO2TLzBsLR3odGheqxoZ9rh9lloodQ==","signatures":[{"sig":"MEUCIGpYMLvcrEMzJs8eK6F7ynWtNQ15i+5zrVsXQup0tO/cAiEA7E6WR0ZJMFk8Oj6WltxpJfQME6QMqm/1dKUmc8XzKDM=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":125836,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgWMeZCRA9TVsSAnZWagAA4CoP/j5xTURSEeMQVM4Do2CW\nOFea3hmWn2N2gGcm9bVkXVEqEE6/fDkVfTJbvR63RBsRm2pfBBYpm/aK4pQd\nLWn8qSXe2ipR6+1dMGWR+I+zLDUWYai77QjHDNMZTXh9Tisr/E9w4UYN4ndF\ndeoGfC4qknI7TqS/Vsb8vmvRAJNYx4a2OBEwp6ldPiDRnauyGh8GMsdllG8K\nrd1d6bhuQ8kSgAODbqlBD5YxQeZFeA6T8lH1thbRTK+NJMma7JgjI82vsJd6\nhN72vx8UeQnxfdWRPsg2RvRD7dEh76PBVN5Pf7r3dlRIq8wWdvN5IUTJ33nX\nO45aAuSpJlaWTUfjG/EHwcRlnckTJGJhtJoqLQJMkwuSjshgZDiZvzYG62sf\nzxjuAwaJ0kHaQKvi+NVYyv2QAk6zlFK15njqwYnMh/BqjRm3jCKRP8PGteTd\nyjcnpXTDnBBj+qS3GQVBUZA2bddvcReMQN5Hjwv1fHQfWZGxFu78il9ywcma\nWds45gvcOxM8+glAOyxNIdJRNs6MgRtmq/hmT9zhdX59SqyYGCVgypGuEDQL\n5fZr/WqkHt8UtNjzfY5hyrZ3o4PQ8GH/3plDXIXHgLnphy48gybGseW1Bm3n\nYs1N7vuvqmYIFhsgVgxRcQH7KwMG4CmrW0qZ3+uTFe2LULaxTD/KvEVepXx7\nzgnN\r\n=RmkS\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-beta/drop/microsoft-bf-orchestrator-cli-4.12.0-beta.20210322.314475a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^1.10.0","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@microsoft/bf-lu":"4.12.0-rc0","@oclif/plugin-help":"^2","@microsoft/bf-dispatcher":"4.12.0-beta.20210322.314475a","@microsoft/bf-cli-command":"4.12.0-rc0","@microsoft/bf-orchestrator":"4.12.0-beta.20210322.314475a"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^5.2.0","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^3.7.5","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.12.0-beta.20210322.314475a_1616431000479_0.32131230807906697","host":"s3://npm-registry-packages"}},"4.13.0-rc0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.0-rc0","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f41d38df39c0e415712d55cddf1e1a4562452074","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.0-rc0.tgz","fileCount":33,"integrity":"sha512-/Or+l7in9Y8NbVLkUSJgMtBB5iB0wrzZkKrkoj0rRWLe58syYX4InFpiHniyYTVBACJiYD+0nrbFDpySzIPazQ==","signatures":[{"sig":"MEUCIQCJQGfULOK9h/imidNoLEhfnVDpcvqXi4N/BriXsR35/QIgW7Xl903nejidQsJA44N3lgq8hir2xDedCqRwekKWcgs=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123038,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgZmptCRA9TVsSAnZWagAAF2UP/3aLMHn6fCBRqJWGhI31\nlWjBHNRScpLu9c//BEh9QjzeJQ+JzWPt3q5d/NQJP5H9UhQZuP9un+UHsjbC\nhbd428zGdf2I7Nnl1xlHjErmDnQQSuOSnc385Hl3Szb9PrP9kCFZP2hX4jTt\neh37sD6KAhrACDWggU5JhEe0ipfQrEIu2c9CMqm0W/dFS4iS8zL/8pwjaVsM\ng5X78Sw8XXe6Xgb7h4wXhnw9TJxAxTG3+IcSCMoeI9SCOQ+aKnavFN1JeG/u\nHIw6zu3UNydv/UQbq+7/uWCfABsjsP0AktZOA5q4vKFvfwhm1JKZ+vcC8mcI\nhIjlrwWOQY2nCii3HPRfaD1ljiGz3Tpwf7JVWFVxd/HaFAGuJj30J7Jjp4k3\nWjLC1MPByJ64GG6nJrQ6jUCn1FbkueBIN5xL+jvBV1pOU2vf1lrwlGPJVI8e\n2Mlt8l4MY0kDOjzEmZqOvf6W5yq/FVA6DtnhpOg90Mb3tQ2RgxC1f1ipIJvO\n5FPSfgtQ5W3pUo334eG1rk64vHGZ2L4PvI/XnJ0qsU/rXZREu70Yf8Y19mGP\nRvV6rl6kd/aCcFK6z+a1+g6UCfg3Q76NUlCitYwT5bgqBfSl6Y4aIQpTBsF/\n8jKVK3Aw4EtLmMXmXW4aHMJOpsxPC/VrwbfT9I8U7gV0KGFGnoYdQzaQI5P5\nMFhu\r\n=AQMj\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.0-rc0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.0-rc0","@microsoft/bf-orchestrator":"4.13.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.0-rc0_1617324652893_0.5811939618030431","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210402.7e5fa93":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210402.7e5fa93","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210402.7e5fa93","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f8a8487011fbebd1314cc50d3107851e1040de92","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210402.7e5fa93.tgz","fileCount":33,"integrity":"sha512-UYzfqprLUPRGDraTwAjEB1Uyg0PqPZm5amRv9FnjB/daO1L8mobztcBFe31OqnlCK221aSaf6ReL6hnabhTglA==","signatures":[{"sig":"MEQCIEIKxWP7tQvh5AZDIIKRpVGEXrax/t7T5elpE05nxdmQAiB9/VEKD37lBk5lDhJ13HADh92ajTeB0Ze7UR6Cto7C2w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgZsYKCRA9TVsSAnZWagAARoUP/RQTuVys2fIYOMasCbNo\nAfpDqaQXxBCHaqR4MBHTAiAf/3oK2oZHpRg0t92mf6crTDvMlFzfl/6e595K\n9DoUtj9GSOzpYF6plIgiZxPp/TGVZErbDWP3gg/2QDa5nTxmZPuHTDbcrjcn\nloInUFDVE1AcastJoTjyAYiYftW/pMGmRI3hqnu5BVs4Yc7IVF9Pt7Ov7yvo\nIk+HWCHRyrEm/usJT9Li8BB6q1i1SFPj7R8dKoyi0pJvbxAhGTbVuoivm970\nGTVig2RQADdF+egvqjary23MuAGMV4/4RP5ietagoig6jsMTbA9WoYHKBn7G\nZBWd+2EETLeTsgDD5tT+Pqb6bq0BxL9UWSSLxhtQsKPHPn3/72IfdGXgEqC9\n0Aq3wzwsFUYmsTUXJz2kKmZ7qiRgWVZ1HittuRN2rVf8sJ8NsjkenPmtpx/M\nPhXrgHLi7dZmTEw7fFbeR3GSlwHNoa16RLiwHgJ02snwKCRj598+L2w2LMjE\nFoubOeFTimfZaKTIulkD2hY+PWEtWPa3R5aebbwQshv2l4dx+Ccf9GEfUUPW\nyZf6WVQfY08btEVn5EXPdHvN2ivYIyIZNgPPcdhw5WQJfVsPkItH40kXuteH\n7WrbdAlVGTuiE9ggFQanN9aUE/vVG6hyAUI3hM/oTBzRq1vlb0DeLuAcdQJi\n82f9\r\n=tTZi\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210402.7e5fa93.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210402.7e5fa93/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210402.7e5fa93","@microsoft/bf-orchestrator":"4.14.0-dev.20210402.7e5fa93"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210402.7e5fa93_1617348106366_0.47238476502689375","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210406.74baacc":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210406.74baacc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210406.74baacc","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"349754a5eb598e4203b01a99a5ea1fe4b4f5f8de","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210406.74baacc.tgz","fileCount":33,"integrity":"sha512-ze26s6LeaKla6h528C7FGsjPY/7GL/1o7tfRI8KJIQPLaFBAqf3oGtiqW0a8snSla2Kpa6coMuK8g0utcu6nHA==","signatures":[{"sig":"MEQCIE6WAJSeYakU2cpwjQcrF+p6QNg6bttqFIXH71OvI+quAiBsLJOpg6rLl7iDWsZOAdAb1nAItBHqUeh5uEOJFOO0Lw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbAu1CRA9TVsSAnZWagAAJZoP/RXJAlx4ugpZ66NWmxs9\nzz/xqQ4B4ii9HPP9GGDJ2vDnE52LKwHl+meBNWkp0MwFteBqcBpF4KJgkVAW\nWuTbGtt+ZdswCEmsGTEh5fv+PX8pxjEmFz20vsEWXjwFvqVXjQVlfXF9mjz/\nwD7223nMcegNdqbbByv9ACjoVvC/WCGEbnIuxCU40Lx3Qb/W0lvUF5yvfcWO\ne/8xnL/NHfUxKa6QzajgP2Nce5ffCruzX65jZ3awEhM78VbFK208PF40EpBC\n0eWumriRyCLR0um8UzMVwR57JxyiczED8i4DKIFKj8q7ilWoZDDkD0SZxuZD\nAvWKgfUefUOB7s3NFRoW54kKDeVxqUVz36EPBTIZjPAiFyPE+VKu6AdNCnUf\nZN6cmpH68sjbrGEXrLqpuUDF8D33cfhVDPsg+6hT1vHVo8u1/uEaQUN10jfn\n61zqhwfH1EYZbzHaG+gTBmyJzYg8Cw59X7sZu+SM0zMad2MZQFhwj/dQIUDp\nAsnuInJbgtApsmywMvMtNV1Ty8iaamGP88nXf4e91luGYMdXDfmjyY49zaaZ\nsUmuW1f4CrxdRbc3BFURv4TPZFbyl7Zkji9G+kfJMSvxRgEBCM4jvr9iwvwx\n91TxaKTNDfNSACllrg8jCD9nvUwVXvjOCTyyMaZ+DSwCpKt4JDJ45PZSLjoF\nd7sI\r\n=ght0\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210406.74baacc.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210406.74baacc/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210406.74baacc","@microsoft/bf-orchestrator":"4.14.0-dev.20210406.74baacc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210406.74baacc_1617693620963_0.09664673798898615","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210407.6745788":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210407.6745788","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210407.6745788","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"0fd286dfdd960a700c05da273bc18426e3329689","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210407.6745788.tgz","fileCount":33,"integrity":"sha512-p0rvhffYxdVuKg+atutxUyfipRq6nb8wClH9yes4DsDna+xIjVJ7yS9Ju/wH82MlKZl9Gkb9E4iQhsrUoJrtdA==","signatures":[{"sig":"MEUCICiwkNRRx5VmjNO39m8fUePf4m69Hd176bZPuL3tyX04AiEA1LzuJPMp4Z8qP9jkxmuIZSK3TJEM8J7KIZxikiUbMkU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbkSRCRA9TVsSAnZWagAACTgP/325+9U9Vlv02jMsTOmR\n4u3ajTSevIUVZ8ONhFQnErYaXI727/UaN0fFkHo8vcvhC5mCOi6/Y4XZ2Lsf\nQ792oqZFdGiT10SMcoq22oqJNNg/x5FLa/dvRl6KEFoju0cUB2DeQpAIyilW\naN+F9mUHNxTngBOI4/WVpRBSxA0Iy9WSGfJhZfcJebxVOZtBIwjQnGPml9L1\niJ5549zx3MsivpV31ityPgjIkHyf8SHhi1TPLcV+ALVHkpQMksx3am7yzHAw\nXlCOZq3IWMtH6w/AQknOVjAC8L10kaqJeB0IC3skaLGDi1NfjivudMZHDBPy\ng+O+CKxtlJB/Y/CShcYETmf82h68/iWmN79WIrAlPAKTXghDryRiAcOcneZD\nBH/ElqZJ21I+qdLBgJ1IlK6f1D3dKOG4UJGz+lfYMd8jPXfskQ8oDMUaNzeZ\n2HYLQjtdWH1DfjyL5LZklFqy6fDw27dBTwNk+H27exaFyGWmlkG8Q4IXge28\nJ5hzBLfS6IMqPP4mpwwEOhOLHFIFIbrRZ5r5ufv8f3J/0AMHy9XRJK40EFoX\n5qZOGEDSwHjihUzmKMC3o4kkHUpup79QLu9oxYw6AVLycQNkAgDvcYld7Tv5\nrP7v6+ubYU/xfAOhMx+r3uLqHpWHJSaIrbeCQJkqelJitJmbpxpZvJ9hB4xb\nDyNx\r\n=DpOo\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210407.6745788.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210407.6745788/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210407.6745788","@microsoft/bf-orchestrator":"4.14.0-dev.20210407.6745788"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210407.6745788_1617839248861_0.9730744436852083","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210408.393d33d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210408.393d33d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210408.393d33d","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"da01f6ae893a90b06e4360467dad76ab6627d470","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210408.393d33d.tgz","fileCount":33,"integrity":"sha512-bIZKpiT5wdhKHLGqxL38EsptJQfiOpMS+89ED1OFMIZXswD2YwCY+sD5ae+HQatVvrkxD3Qzs8+90MDzZBSXXw==","signatures":[{"sig":"MEUCIQCJh50j1JbBG0GPa6iql2hDqg6N1WnjaAfsrJuG7iCq2AIgZ9MCsj4aPyJYiZ5H7Lj4jc51CWMrftrrFjtrZlnuL5k=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbq66CRA9TVsSAnZWagAAObwQAJwYPhdjoDTqPzkrCXoq\nWa3u+DXcd8VtYjO/rentu7nMC6vUPj8aATFjP5Qhh+j2UxNMI1fX4iYU/OUe\nz44sQRYQYVaO5PNp9VXnB8yyVpkEW1aZ2WXSGAneH6ZngXcbK0tuijYaLfFU\npqoYzHVsPybpWm6rOAMHHLFCtKAgjj2Tpf8fuQOOR5mu+ltVnf6k0+U6WUsS\nlmTPHzljX4cfAkkTMLzqHGWqySMPGUd4vnv2N9kEqA+KuP9kOKhB/LAfkKKj\n6l7J4MFM+wdZ7erwmblhlnoVA7rX30illfVJ1k3k9Sc7qU4uMizFqe8ipemj\n8mZKOs4cF0NGK03OivSPYzPQwB+e2R5U0uNLa9L70t7HKHS0eX3oZyiAQIZd\n69FxejBsKO3b79gPn+2sU+xphGCoxRL5Kv7/iRhKsajgwzsJb9+Ojse14xfR\na5ltpGzSh9GXvQybZtayyfcLrKXUge9zJ2lMcd6umoan2HR2iIcIWHGZy2si\nKdQhdjHVrZUzEqw7PF9N6SGUfQSqrq+pzzH1m24NFubXh98ySSbWt61We7r6\nbFx6hO+GR55ZHUAb2rhfQwGohREn5demQH0Mpsw36gsMPeFDYeOo4eXTRJT1\n3Yw6G3Ashyf9yiMQZl8zR4DDHvmepY6BmZlX18l9JYEQNY2B2eFKwiJ+rmrk\nj07w\r\n=p69E\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210408.393d33d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.393d33d/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210408.393d33d","@microsoft/bf-orchestrator":"4.14.0-dev.20210408.393d33d"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210408.393d33d_1617866426238_0.7906411168151009","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210408.1a5070e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210408.1a5070e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210408.1a5070e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"ec181ab4f44204841a1f24594d06db6189cf3f3f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210408.1a5070e.tgz","fileCount":33,"integrity":"sha512-ABf+LOm0STywSXn0TpA1ilIRnsmgHr6s6xcTPUHrPdEKs3lqkQS1R1cegEdc4eVzk207rC4QyUL0Onf87bN8kg==","signatures":[{"sig":"MEQCIEWFpwRoNhJjL2CWt7A7nWgZQC4g7B9dFd5rCjz2dvLnAiAkF5eir0opWMURI1n8QOuMd2EBEZN17hWx1V6nfc9O5A==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgbzbWCRA9TVsSAnZWagAAdx4QAJXx77Ojd93IYR7SaBAo\nsvAwMrAbuYoNtVeq6OBI4RdJnOG+rUTQiNrVYzb3wxUZwsGD5Sf6BYviO6H+\ntGT3Jg+ZURM0GhYuNtXaJK5rCDhFsurkmWw+wKrfgRgnX39GiJ2q2Ni0M/LH\n1NwGRmRWDqyWo1NzDMzMoVO8H+dZtpx6wMFWhq7OSuk3Pt5NVgsJVQCVIIWm\nfA5rQ1Vqi8FKLJgOFgOrtEdGlWtQAABvtkvSDsObbR5F2FT1ZBPdGeZWv6BG\nx3Z96qm9ZAiM1hEhjerXkWLkt10oXxCIIILYdTufLysWP8wICnEhsa/ymKc0\nOrqemdWW4Vcvsz1EBBsblvhJ0A3HyhXMaxYBYZrSo8W+Y879EzttxjFuQABQ\nUoCSQ41Q7efmizqkQfjp7ayIAKCaJz89wly5gFljWO6M0gGpFl9eMK6paTtS\ndH1fXUXUPX7bQSLyXGU1MmJMF5tEglDwlp50e9S81dJJZlXkgdU+vuDdqLFp\nWY7YAQwzmeaXCNFdVCxTp7ufotPVn+99v/EZuSsNQkxWZyBNqRdg4D1FbVLX\ncTPkJCEawtCwTrYUQ1lNwiWeDPEookejLzkT2BKetClCptThL2FmwxUKZs3z\ncC5EY40RYhI1RBbzjn1DKhcFbHwfoWmlhFzune3YmCWwMlo7L1F2FZBw4oq8\nbQXq\r\n=v8Ru\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210408.1a5070e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210408.1a5070e/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210408.1a5070e","@microsoft/bf-orchestrator":"4.14.0-dev.20210408.1a5070e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210408.1a5070e_1617901269942_0.5662036646610784","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210409.1a5070e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210409.1a5070e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210409.1a5070e","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5580bc013e2610a5907d81f1068b47e3087f41b4","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210409.1a5070e.tgz","fileCount":33,"integrity":"sha512-wz772XoglMNpJ8sCjz5ZoMZo3+tGVjHGVOEuDsSNjmyRcNYViEwKIKjEdPepXaO+cNNtsJtthK6FjbWegIcAoA==","signatures":[{"sig":"MEQCIGzP6eWZZrBGPD4hRaNtryAkJd7D906EJThadls/SC2dAiBNYSL8Jr+yYtXabninWFZWupdnbaWCa8X0KV990fwZpA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgcABWCRA9TVsSAnZWagAA988QAJh6kvs9Ihr50f4kYJ98\nb2bTIVi3rfaifHBF4E3CUF6FA458Lc1LNgbWyzVLhc20C/L39jpVjjWvZB68\nraxsmX+csrTK/RgpiWbeckLoF7lR+z1xeBsmAFWdHR5XbMOiztkkHegY2xzY\n/OetRwyyJN6Zz8eGHhaN91YzBuY+4dCsHBdbb/JX9DrbeTL6Ri1kr1CC2T+y\nbxVl6PKMO01TBGGyKhq/U/p60EGRTXgE37gbB5tT4SHVcVmSvq8MeuWIrrm0\ndTMeI1afOYkoCL6hbRXuN7wsQDOnU8lSn9PghogdBFa1cVbHQUxdmTj00+mR\n+zlD8jdrGrlmV6+Xb1h4qusTyXNfjbp9nG+UOTztlaN1SicWn5GVUqNMnpmi\n+zCbCW4DS2+q43VTOLxf3Mp9zeNiMXvK9QYzza0JmKzNsnJH/oDMarZHB63p\noFHJZ/DiKus21jxUM9gXYuQ2Fpdkw9MVyUJgsOJHTF539g+3HhUQi+TOrRP/\nV+yz21SbvrX8BKJfikaJcPb7mX+xY9ktKUbYDQ+vshSS5FFoN2+yHpr+lkZO\n1eDOXQ3OIqGSqVAco3279xVO/ZF0446uP++ohVZCFTKcmHsZgWh5q9QqqKgk\nJyqAJEIfgpmhz7N0dbTG9sDNirnUo6Bl8HAJLU+JqdRnArGqNykD7tL02vrC\nCI4s\r\n=reHT\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210409.1a5070e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210409.1a5070e/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210409.1a5070e","@microsoft/bf-orchestrator":"4.14.0-dev.20210409.1a5070e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210409.1a5070e_1617952853769_0.40256797789188914","host":"s3://npm-registry-packages"}},"4.13.0-rc1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.0-rc1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.0-rc1","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f77e6f9d623db1e0e369b7a61aecdad8591c0134","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.0-rc1.tgz","fileCount":33,"integrity":"sha512-RZ0bXtWC5KKn2N1U06PCi2IT9QN2JcciHcW62mV0twEcffIRMYhb3kgZaR5Neg3q4ySdww94szWY9xcQnjHUAg==","signatures":[{"sig":"MEUCIB+bHd2j3y6DxpG5PxzPFC9QgcfedEE/3bX6HW4A0AyRAiEAnDaU2eXFF3/+5sGjNDCE2UNzzpy+mooO5FUyrI62+yI=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123038,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgcI9JCRA9TVsSAnZWagAAqRgP/1SrQYLeyhLijzjPByST\np1UaAYMfiIk96hssSvSOdFoXKYRG9ykXjizGEiD/KCXgu2H8958jc43E6EaT\nXnrlPSp79qB+dOIZXFfofqr+oK3kZg7LK2cQuFyXtb3QuQEIvNg0sXxcxTcn\nxi2078EkYGmkHEWpPK0FQa87ytohCmlcjyDtfUFCbHG5TrB9ont6GIHc+mHq\ndziALRyew89/b3oSMKG6XoG8/bMt3RgJLmZRnPSF15P6s3cRZqoh6J5EJthJ\nfWX1H4AD/XlC+sY+eOmn69FrDEAuDgW3TQ0BlqB+g2+DXn+Gx/Ddt8xRTKHo\nr099uGXQ6pUCskjthpBkkU43WStQgmYhpK3qN+Iw/JgEn9v3IBiUHnvqN5CE\nPvSkbu6zqhHd+XjuaANQeEJ6/5BdzEJrfFyXlIoeuaFiHq4guxVLYZCeq887\nAxZpoL/CV5UDypUZeRg/ESW0ztxkmXVcooGb3qBSeB5o9jEAdcwkM7L3lcsj\nZ533CrNobZzRBPiDO8cxoessAUQUBr17bWccAUFOHAuuMttrCrmfYa9j9JD7\nFiAowsgU2d1TZwKTF8ejgMpW+kaiIA0E9jtV8/OhUtU1DeoSEOSYFp+fHS27\nCLYiOBtYW1qm/TVgT4vkwTFUNZUAKXCQ2keY6mF6RliojrNbA/nB1/3jY1Ii\nwsJH\r\n=MvNO\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.0-rc1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc1/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.11","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.0-rc1","@microsoft/bf-orchestrator":"4.13.0-rc1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.0-rc1_1617989449298_0.3089812724384464","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.161c029":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210415.161c029","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210415.161c029","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1ab35426fa0de3e21f13bc05c845aca43b74c087","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210415.161c029.tgz","fileCount":33,"integrity":"sha512-JBCirHoNVR3ZLqOgCZ1n5K1KJrSGJ7tfNbQI6915NWZWo4kmWSo3gOFHg/1BgEweLos8oq3EbGX22Y/GZZozPw==","signatures":[{"sig":"MEYCIQChyLbqSdeNYFnbNZUPmL+B9qJbzScEC+dap4POJJnXUgIhAIfMADKNFrCxT/P6ls576o79ZrWGdBVN4iRJqiap1k+l","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123259,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgd+lECRA9TVsSAnZWagAA6I8P/jaGsXIH91lr07fIbP+I\nKBuSlaWGuB0uLNC7fLb/Q59Sc24CbKGS8Ydd8ifMpS0RdjxGdc7S2cqHzgv8\nTkOiwtaVK6zIqHanefE2UA3aiAtPKl9fj7+ERcPlyzMzD6gtjKCvauD59WQz\n8WTaFUjx8kJEJ8wG8uwLAIhWpnkZr2ozt0mDUd8ZXZk54yNEFBzs74C65GTM\n+NLOyinJA1cOMQtZr8zltVHMVQnN9MycaQvhy7QByh0llf4BnEd35ROK8AYZ\nf7f/tY4Kcu6s3sk+7d4YqNK0NeHfByuxFE+FwYbwOnLfCf7EwiwkxcCXGME8\naHGpjYx15rkGha3NMc2NyRDoOcBLDXpH4wvi7U2fek7i2YwadY2RUKzScmYO\nLg4xT3utLksiHaFkvYf/hj1DZQTiGaxvSq2aQzPZS9/vrvd+LqBJh6sH8zEx\nMhBEtJHCCPRYBE3DCrhZHJhQOowLvJWzth4aXQQ+CBfhi36tmXDV1MYUCp/1\nx7fjqlH3dUAtSz0zHIKj82dMd4ufYgjOfJnM2LMjUhEh5KPHMudufJv5dXGR\n7uxHCPwOuh+LMxC186R9NkmXmu364fDsmj64S7Ftt421QQLPwxbnK8Pht9l3\nosUj5BsUhJppEiOdXS78p4PVapPk7iOLBBXXEfCmd+BAD6SpbFn6WJTQ7vs0\nD1Ho\r\n=PsmN\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210415.161c029.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.161c029/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210415.161c029","@microsoft/bf-orchestrator":"4.14.0-dev.20210415.161c029"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210415.161c029_1618471235601_0.5840587342402153","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.7e35288":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210415.7e35288","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210415.7e35288","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"a842186f956841f99e42ac2fed056b9eda520ccf","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210415.7e35288.tgz","fileCount":33,"integrity":"sha512-DrjXDNfJiVcmrNidTeEH2lNXRGyp3sBAqwSo0WM/OXOwUoQBZ19NrpJCPqngOH40UQ0+oe6J30Jznr0O+CwjVQ==","signatures":[{"sig":"MEQCIEDwJzs7S+hFtPn5tG8Ollhsm4MSQ0aXP0NSgMeyXDLKAiBxG9+s9HoKqbj9ANkc3vOjYuJzUPoJkk+Racyu3Oz+Ew==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeJDHCRA9TVsSAnZWagAAGC8P/j9jahGFuc29CuwWFxJ7\nGD87/9+yyUFxsj3sM4jYHBHSswCIFUy+4XHcey4Av1+8l1IMua20HhWPWO20\nQ18ao43MvIfPmNXMd++rdi9ra4K3MkgYHyx7+Swmjr/FBvADQqvocf0kCJls\n31SbrZtnh7TxFvrUsfap0LxbogW8mUK6J35MetTUw8DcB3iW5JyjWmSJxQ9W\nPJ89AtZIRWoqIFWeoXkdbwJM3T6vjeYTbfAtAyQWx69FunfWaKp6FzDL4zcW\ngnHiricZ+4/ABtXklYck/m3yrltdPd7nMxBdQHZ1P7n1f83ja2EmQof/Ytnn\nuTIk2JCoW2uZzHsz3GIcmxHzwZd3s2+5T/26vZW74hEXCoL9eJIDdmSIkWsa\nMIIsKcoNsMe7ENL0X8iL+K0SGzXlQHVgwgKmkryMssb/sb8X8XQWIWE4wSmM\nGJw4LXWMStUmKaPJM3GE519clUEY65wLHTRRFXL7fZ1BgUrFCAWYxZpU/tvx\niuLag3wXqMWE6GrHoZSckviAXo76kYLbI6miiTxyz3o6VNPNS63XvfWf98RE\nKnXW/M/KP/Yb6bxJobRbwU0Cfd0WtrO/CqKRCho90YwMpJJ687+KNQD4iQLk\nYpRUtH2zb/ovvgFBMNdOpG7+MxuOWugIWBrZ6/nGF8CWVzO2qqnRZvaDBptc\nTf4p\r\n=JO7e\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210415.7e35288.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.7e35288/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210415.7e35288","@microsoft/bf-orchestrator":"4.14.0-dev.20210415.7e35288"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210415.7e35288_1618514118977_0.47133749164598826","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210415.ab0fc09":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210415.ab0fc09","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210415.ab0fc09","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft-graph-toolkit","email":"mgt-npm@microsoft.com"},{"name":"darinious","email":"dahoove@microsoft.com"},{"name":"benkenawell-ms","email":"bekenawe@microsoft.com"},{"name":"abrilgonzalez","email":"abgonz@microsoft.com"},{"name":"awentzel","email":"awentzel@microsoft.com"},{"name":"eisenbergeffect","email":"rob@bluespire.com"},{"name":"janechu","email":"jachu@microsoft.com"},{"name":"nirice","email":"nirice@microsoft.com"},{"name":"ccf-bot","email":"CCF-Sec@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"6f5336dc017e1a55b229000f1b4517ca8dd59e7c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210415.ab0fc09.tgz","fileCount":33,"integrity":"sha512-ICrAoMSwZUvMDvOB9/TaXE1QfnHPTEYU8krqK48H7d9HzV2iQTPWMkqPqq5Kajx1CzqQlATeqsNdniUo9koWJA==","signatures":[{"sig":"MEYCIQD4clZJIDqFxLw572ul9KLIppZC+8ae2N45hMsUCCWSkQIhAKsxZlCYBCHeVjQ03ZByfjT/xKCVtIPKNetFoV9fdaXX","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeKDMCRA9TVsSAnZWagAAAr0P/3gKK9hE0yjt6o1WPhVz\naZ60GX8f90vPmcFtijcUX/LLasAEoEUFGu+cyCdFDWqBHColBLirhIn2/l2Q\ntfwjv0N3F8eV3if7gIp4hnKMFYBqcpRblS/i+Tw3BF/jWWNFLVvVAJ7jTDq8\nMCP8tukNGMbviNQT/OZ/p3WdN9Kp+tAyXBEkhc9SERt4K8IXep++WOfHfLdo\nvYKcdr3StAcGQWZkIxFQjIvAFfAW0/UL/nP2hYSO60yYHQwmkT9zPMCnXye3\nxJbz8kUv7ATGIxIC3S9ofI3sKDkLB8ihESLGRl1ugjSnOABVF6NZzDT43fSF\niJPJWyRnPJmUhD2DeRiJqYYty0xi7MEN4jt4XSSRiMphykwFyvmfbuQfMgj1\nsslZB3391bfhFGvvFJLG6NkttcXAaDtC2m94LNf5wIdgGYzmt4kAuw7U8Zrc\nH1dW6F1Z86SsCZLAiGCmETe0h9BWFORKcBFedfmmJU2XlnXMQXy+uun1AM1q\n9WmQD/ZEVN2TDwCTG/8oTVPRnBVhNwUTur/2yXsrzviVm/eCiy+ZEmeyCW4m\nSWbruek0NTyvplmo9qEgb8zBCWApF/xVzGUnsT6wt060zdxf6dEYQy3OpDMO\nWACQcZghq97tgKBFWHPBxZpTnNo4uel5ltYKA8Zu7Etrlf5obL4SisUq3Gah\nshl8\r\n=Adsq\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210415.ab0fc09.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210415.ab0fc09/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210415.ab0fc09","@microsoft/bf-orchestrator":"4.14.0-dev.20210415.ab0fc09"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210415.ab0fc09_1618518220168_0.15839428105152975","host":"s3://npm-registry-packages"}},"4.13.0-rc2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.0-rc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.0-rc2","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"462759829d2b7dee73a9cfc48ae789367763ad52","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.0-rc2.tgz","fileCount":33,"integrity":"sha512-7rhk3lMovJCQZeKWg/fr/NN0HFvOJugQ2lnQp30yRwrPBGTrZtyeG/UW0FTrg2oveV1VNeBrT9Xgfoee+KGl5w==","signatures":[{"sig":"MEUCIEHGSkT4FvrD5h3b/cSffIM4BBF0DZjo+7CwFuBfS93AAiEAr6Y0dS1BB4NauKOK13ndAj4Ydam9mrlkGsNAO8j35BA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123411,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeMNGCRA9TVsSAnZWagAAvP0P/3pjMk0M2j/WzYgCtUaq\ntLRuyAFUoDMedqFklKCEhW0yv5pVEnaqib7Vlejb4RZgLnQoqIqxYDzDG21N\nsPk9hKPixDA7Y/nxAxGRAmKq8ybBvsJOfHNRXIXkHk4l22463dpO/WhkT8Uq\nq1SJGbzA593/QhNVe+JZITGiv+MoNyOo9O5csILGMt7orGSoB0WPzEa1fYOw\n1hKIupMXawV6NdOVZrFGCBCjcqyKUDCr7O5PAO+8BgO2GlY+rRfAoq7w7JOk\nHqY9O3U5X+On3soAwyoHPDODaIFWJO05OonR0THY2eBzbWuwL5C+QioaWIq/\njjiD6TVN3V1IX5fh8pQK5t6je6pX/pdHWwe1fP7jGXELh2GZ4lmyqB32oRt2\ngTSFItHV7zBdO7/boFMyBvsWY8jYe6PClAfqxlg/f92J6tXc/wswG8AuUlpE\nfT2TqEvX2pVTFwzDl55GWKumBOhmpZ6OFm+06Z+vIrqfPQVyH0TzEU2fMKqb\n42MPlvi/YZO6GOKEajEw/5CWwiOvJZ29PuSeEEohY9ioX8SR+vRAKLmcPux2\n/qlopfTzyHhhM3XqdqlsnYMBGz06OoEzfUGw2Yd8cMnG30nkNV7AjX/OC6fM\n31aFhatFNz8SbeeAD21K+0uOF270Hi6MbxrF4NYsnWuK7hm7WdyyWzcfxx/S\nGlCB\r\n=WRjB\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.0-rc2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.0-rc2/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.0-rc2","@microsoft/bf-orchestrator":"4.13.0-rc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.0-rc2_1618527046386_0.29003351272046496","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210416.ab0fc09":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210416.ab0fc09","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210416.ab0fc09","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"ba2b1ba46dc060cf5b297fcbd637807864fb91df","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210416.ab0fc09.tgz","fileCount":33,"integrity":"sha512-wCG0RD0r2FnAD0UlMaq1lU4paMjIPVyNRsKB22AWvIHoFYh/3RoKoMSh9sY0bbikG/YD7RlsqDil02iT8iQuCA==","signatures":[{"sig":"MEUCIC1/LFuGqhW/mgiQkQx6s2Ml/ZnccR7IfECgACJ4ekwtAiEAujAvBsSinjiU3obhb3sMgtz1FE5505zNmSvRVE/l5dU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgeTrFCRA9TVsSAnZWagAAZ3gP/2SU4dMeIHzDiFRnK/SC\nbnJnBeW6012YMMl6D3Ncqgz+i45VLjACQjSf3U4TyF+NQy/ep2R3U7Z9+4Un\n9u5PKJSUWiO2egnT7yKzDQ8f82lCMKCK8evyjPwlXjjZmharmEJ73e0dvcmJ\nKsCuqJtJ36CI7JKmq01n9C1OabJyYy1vChgvXp6goxZ3Nbq75Cwi1wuiM+xS\nwhwZLLvIIdf/9E08En76nESRAh0hZ5s69+pqfNIFziOefemP5U4EQW1/Q2I/\nZlJlMK/txN1txI6JbH//yf2QcwJBmcB7qwsPaTnhDbBel88NonTEVeY1lmiT\nQW38G8ZtTWG+aV2Al2Ux2NarLnbQ9Uevfik2ubLS8HbLyk0CmTHvZ6cJQpt6\nbg7Z+fyoa2EodzmiApmJKFnr0fkMLDOV0wLU4Rb99xi8ugE7KdjGZRhUKmDV\nfB9k1G/tDTbtBV+E2mIHdLGfrQgzeu8w4h520beDKFbg9RfpIVAuwg5OQg5c\nXMio2FAln18nf1F6DZgg7OXOepq9igaO0neiZ73s9K41xpZB6wMoZmVPErRH\nqaXotOoZvN1V/el3vJBBKdhH/96/613v5SnBj2Tw5WHyPwaETT0a3dSxExuZ\nal9k6O8jURdQn0/wMv3IRTBjhbKdndNAEEcCihC8A904igC0wZsl1nq80EYS\ne4wk\r\n=TT4s\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210416.ab0fc09.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210416.ab0fc09/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210416.ab0fc09","@microsoft/bf-orchestrator":"4.14.0-dev.20210416.ab0fc09"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210416.ab0fc09_1618557636522_0.5621664149793897","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210419.7257bc2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210419.7257bc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210419.7257bc2","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"58faa921718eb9ffe0c436b509e6fe6fd5726b74","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210419.7257bc2.tgz","fileCount":33,"integrity":"sha512-pRi6Nt2tK01zSocHge1cEL8LbuILTRoFiCf51ti+l7rlRxN1b6Llj329VdsEjmSihzLWmiUPtTQjPygNDYOXfg==","signatures":[{"sig":"MEUCIGJBIZIilUXrIVZ3FDTJKEbs8Bh7jgYV2EnHNdWvBh63AiEAm83mUyvF+YrhO9J+p5ofkqG3pKcQirNsfVWved/UoM8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfc1SCRA9TVsSAnZWagAAyL8P/3GvzRTS1gEQR1RMGB8b\nfh5Z9878DvpElNVx5QFpp/7Kzcbtyo7qyDSbRWuznWlHXK9k05QxR1PJCGSM\nNEXnr70uyl/IimTUjU8f1+p9oKHliGXopU7gdX1LrEUboBNYTy5r/TZnX10K\n/IMgoOG9BOQNrZ1q1eR6ZwqbWABe81a2aYyZwsIQr4a0XFwAY62ETDySGRf6\npXDDJxGoi2S4XzKwfDqAqRkxKkoGeNsSOifVi9qCt5AhgJB4DnERdV0nRX2w\nV1iE4lqVJubw2jmUglXtwtUR+mlFC7nIAb759LFfeQGoKPMQB54lvjetFs9g\nrvqEBCcpiSjfl3SEBYNl4tCxoBDtwWp0HizlqJZ4A3Y7PZZIiP6G+cuXRyPy\nc8BZYiIuERLMZyaqwgd67ZmLohcWj5rIAG8lpKuh3V4MMehCalg3kl3N41K7\nInUW/9042mJsN8qCMHm58uiUUWRISmbF4qQPo3uksMSEvNK9OhoQzQMHuNuV\nYHn9TeQCRpoccD/Nx9GkbF4woA8Rg8IXfA8NRGjI6lGNTodpeImCXgtek3Cq\njWz3cXZsDb+KsAaVWTP5lTdDJxQlZOuYx2UGu4J2/55kRdnJ56T1SH3zATBG\n0OyOEA5EDB67P+9eH/Ajmt8DFo9BiV3EVQvaA9ZSZ4GtwLg/M6iCUva6+gp+\nso9X\r\n=bVzl\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210419.7257bc2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210419.7257bc2/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210419.7257bc2","@microsoft/bf-orchestrator":"4.14.0-dev.20210419.7257bc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210419.7257bc2_1618857297551_0.39981389600089345","host":"s3://npm-registry-packages"}},"4.13.0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.0","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"d83a69c1cf8129ac5ba596a71bf778227a8bf111","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.0.tgz","fileCount":33,"integrity":"sha512-TthXpbKt/k8VmdKnQY/D2iKZWuCxi0vjXJoxjHWOD2FgTgz3nEJz4G64x3J7uL6QiAtXPxuo62/2VrC3ZcGCNg==","signatures":[{"sig":"MEYCIQCNbGF8ibWCJAAPDLI06au5XjtRKPLGI9ljI3T38/GYwQIhAMwN/qvJyn3AZLP+vdfYz+63NY1ywkZXma8M9Y0jufVO","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123359,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfgHlCRA9TVsSAnZWagAAZ8kP/R0rGdQ8hnqET3yu0NUd\nCEWLmFV+CYwDgu61D6dUUU+VGCo8k1j8v0Eduw9xnux9nIVi3j7Y7arS8aq8\nlW+u7MpMs11bE3G+YV8gBMcb+zG02j5xMA2zyQh0QE0JFC8Bx+dZX2Zha47D\nrPqK5RmnvPZwCm3+3hPah4am++SCgtymJnfbdWRUAqDsmI+Q7lmbTxQeNeem\n1zuMyW5L+MQ9X7zAZG43fCUDv0z81zL7RNOsmypREUqZ2PHHAERn5d1/2m1L\nhf27PexbWNRyh3qqKVwK9FKcEOv86bFUbSl+Iat8/87gY0WU2zFkjCsKs33D\n5UCD4pfLB0WdDxHsiEcdVVWWtwJ18uL7dU3eerOGy0Ii6RM8BzVTVQQh2xUX\norBGufemzbKd5bqyfTo5fhIyQp8fCP63svNg3N0YjMj100j0z31QBEE51bOZ\ng4cWeirR7WKUwK7YYLk5N/g+uHRAu5tMWRWWdrAD87MCicskqV0h4buOjXVt\nSpxdRlDY7PGg/QbJvTsPGvlqpM6h1afbLF8siNTn63AWq0zMbuItY/uOeK6i\nv50VI/0Ntr1filMJycEuGNu0YSNE1fl2NRaGW0HoAsDLeNFD5ltcf2ILso19\nihYzql9cc7V+WxSYQ3WBdZ0V+a4S9bILXkxN+aLd58JSeCmgADc+KEtdpbXN\nQZOv\r\n=Ml8r\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.0","@microsoft/bf-orchestrator":"4.13.0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.0_1618870756610_0.8892924093709555","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210420.fc92e9b":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210420.fc92e9b","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210420.fc92e9b","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"zhyuer","email":"zhany@microsoft.com"},{"name":"nintan","email":"nintan@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"4945558c209780c59af2433a9246a8bf70f6e418","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210420.fc92e9b.tgz","fileCount":33,"integrity":"sha512-xEHZj5R/eL7/yJBNtVLfcdZ3cF9nAIU9PCmDemZ7BsCluPNOtqSNVBl9VNSVkfMqNOaaQcdmO0P6pDvF5MT6YQ==","signatures":[{"sig":"MEQCICRjQfGjOicj9tP6JvEVzCdnC411lfzXhxdxXFMrzWx0AiBXTWdh+GmFjnDmfGVsb7az48ancrHyoa27OtlpAhFmMw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgfoDJCRA9TVsSAnZWagAAHJIP+wStQJcIwFY7UNXY6spB\nLpMBsMwZqwh1QbVeRZeIh4tNFZFT7F+oFCBKhZOtyQxMeJt/SFtex6TpUs3z\nkmXgzMqXYWCouBivH3vz1aZ+9xr0lkyejFUGiv9UvpndFbhZr2uygULo3TO8\nYCn5h6aBOOg9YKCOyqA51tcjyeM5cD1MTw8LLVyIW6rsqoJ0EojU1bzsiVR1\ngtYIYWmsK77/89NROh1aW/FbkolSkxzYmrxh/4Hn+kO9cNyb5ETxMsazQFqI\ngo49L1vrXR10/E6c1RnI1YjCohMnMbu/yNSNElso3qMY6jR3HkJ5jOz6hYao\nEHs8G5R1ypdKDqVO046mZ4/T5PgQ6VyXjQtcJv4UlkHsSRDg7R4noIGDGTdw\nu23cR+6rgStbVzW9X/nD6C56lHK2hwrGWfdKP0DWYXN013dbPyK81QYPXQJY\njYZ65xeUI85ynIsBJLRW8UbhDsDl6R/k1bDol4TPNjNIW6y7WWhfD8M0OOo3\n+3lGL/RNxLtYefRuh6WoZ8gKhIinuwv2QOm6RseAvCIY6jgpCv6FaUgpauE4\nvtp8BnLLPCgPDobE5hd79qm/hpxhrXI+2cIHkT+sNaNqCAPx+7g5bamRQ+/1\n0owofI2Hrhqkq+oayMjwcwNMmNWw8FqkyDL6k1T5k5RB9jCFQFYni4qLmIeE\nae0b\r\n=IZug\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210420.fc92e9b.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210420.fc92e9b/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210420.fc92e9b","@microsoft/bf-orchestrator":"4.14.0-dev.20210420.fc92e9b"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210420.fc92e9b_1618903240340_0.15297118650666897","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210422.eccdf02":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210422.eccdf02","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210422.eccdf02","maintainers":[{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1b019b10afdb43ecde3ac3ba3d6bf595b65f18c9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210422.eccdf02.tgz","fileCount":33,"integrity":"sha512-Hwbe/Gdt1TKWFuFALm0v8HM7kFbSAQA3GD3O3fco1LIn6dMJg0GiIvhR5mgaynDPkenusAVB430TsdxllK9MOg==","signatures":[{"sig":"MEQCIAWR7pzw+RdycgXyerlAIBr8ZgY5SYSHL9K3ylxGQRErAiB/Nqcpu665CYMef2oDFOmDEeymW0UMyswhGzJfbeE+iQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJggSQcCRA9TVsSAnZWagAAqGIQAJMKNWWgdGT+09IR0cjT\nllg8TNCprbiR/ucdR1BFUlEol4lAo5CFbrbzI3ZaeNSMM7Xr6S5ZQZ5c4Tgq\nKEnDVEPqLSWoSpRAbkZHqtcZ1lI+OlRsGpfRWIM7viXPMAYD7TLUX+9HQqoa\nQF3uk63mQTh9IBLE7fJKaXPpRe7rlbp+VnZEiKOm/Gobq2f5Q9le7fzLBDdg\n8VtUIDzV22x1UBjgK79CKl8B9G0CY8lmAoOANvOSvtPx3dQaDYzDd7eoh0bi\nib68AQexRZzIOotUWCq/Lnaqnw1CTd9adiioz/ZEGcGDkxNw92GQlxmsSlWs\npC/IXlVBi8KRKwGtyWF0jVWUZZPkRke//vLdocx39wqrniXtaroHkxlpzC0u\nusSJMo5S5c4DRvVrvh0R0uWoCTTk6HSmxDD/xqi2SCjdeMxoAhFPOfECAAhR\nQP3Wq4U12s6P+XPm8FhuNF7iIUejsF5SVLxtM1i0fsFVU3NVs0QjjrW+L0Ae\nYqwrK8ayp/8RpEYkc4pYpJmYxYhjtY3B3z79wsmgaa1O1HtfgBHoEXrGTSH3\nDOAV/16MwrmAZ32TJEyN7wvcONAPfs0PRtQ8MqnC83Mw7WZQLFnHRkf5YUG3\n30QRbA9YH5M9AbK9piPSrAt6p2Ogn8BQelJhNSuadsO/X1mbzJlAtGHw3Yrp\ngwsO\r\n=KPlM\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210422.eccdf02.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210422.eccdf02/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210422.eccdf02","@microsoft/bf-orchestrator":"4.14.0-dev.20210422.eccdf02"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210422.eccdf02_1619076124064_0.3975953781846999","host":"s3://npm-registry-packages"}},"4.13.1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.1","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"fc9ff578952555f73fcd4ca608eebfdbf07bea65","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.1.tgz","fileCount":33,"integrity":"sha512-sZRfTi6RzsVJTpDwyLZpwXfcFV97ZVuJJF4o+LDr6kzuf55tyoUo1M1VLypWrpG0SyRIOhUpJGWVjfOQw+hcEQ==","signatures":[{"sig":"MEYCIQCZXR3iqc6NxDzdjP9Ui+YbRbHeekbWFcY58nQERyhyWgIhAN9iFR+kWkSNhNIxeICiGn4jr8+p4mGscR+3Gxp+9O4j","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123359,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJghzNzCRA9TVsSAnZWagAAtEwP/izFDxxOeKjAcXlKbXwJ\nmtW7tsUN2GEfnGIHs28okLSN/H1ceFNAxP6hWiQKb3+DVt3kZGe/hEhZe6mh\n3B3ZRq8FZF96wFp1MHwMEg/1ykgYw6qzI519j8c4J+nAiZ+Kza36g5GnVL6P\nZ6FrGCu+Z21aeqja6AlUp0mUrR2Vol3IoIb22PxBrUpwYe2uerRpZ6SlqfNd\nIp7LLblkjERljWiN+igF6ZJBfW9BnTMoMq24H9TK5BwgtyEanZhxE59PS/Er\nO4l3o4onVHCFutQMxf7EXlVb2AXkI9JxzA9rbk2daYCzEaNQFo5X6pXASSwq\n13k4nPb5w3mH5b9yL/ief9rHVdecXSKfWOKe0sMkjDBBKUoHS076iv3yjjzO\nb+PlAvE6psjqRqoouqWe29k80pqSJ/GcRG/ak9GQXInp/bmVEsBA29wgNjEP\nkwxqPNg8tQhm8xIWah1vmqtHDwlLWosVu1ZexH+mAQFDBiGFFjGMx5zkaqkV\nyxmDHm4ikKjmdzfYKi2MSy7ZIKrrxfXQYdteumjBxljuiYZbKYuEW/NzZBKI\nVRUoa1Tr68t10MF8Thkg9ALAWDmWTcDJ6TD3gmbpCJN6XvpSvoUuQl5J5QWd\nP3xzlPBSqi3BQuaz2ibg/I6SF46QqxG/Wk1v/RayLRwNLad3qFxLFHNU5J5w\nPGXY\r\n=eWnf\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.1","@microsoft/bf-orchestrator":"4.13.1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.1_1619473266681_0.3243408562100085","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210427.c0be653":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210427.c0be653","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210427.c0be653","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"b61ac03e89275a8b2a412d7ba3039bd6c1081fef","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210427.c0be653.tgz","fileCount":33,"integrity":"sha512-nJFHVXOYBryfYxKxwDqHxPPxW4V5m2ZjYs9cMDZS3lK8GAMVQVTQ9K39r77MJE3gUMpccQlcA2pCJ32yyqMxQQ==","signatures":[{"sig":"MEQCIF4csVSQdM0xVVJZmzug2DvTto/DeQMj6GoT5rBpF56UAiBGQjU6nZgqyXxvfyOxtD4rI0Pi1liMrnCjeS3BtE8IWA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgh7uSCRA9TVsSAnZWagAAt9EP/Aur68akCgDLktv0Bdnn\nVtRy/rFfjAx+UODptabEvWWuJ3QMNFyFQ7KZ9y6z0GkKF21xsuTOe31mnb92\n8YFGkz3/6/udqSnMlPEtdcHnsv0g9Aj8lrMaRKFblCOpsaxKdZu679Jy16II\n0BBg6YwllPMqgtyr4WM2IciPg1+rZ5Kqx3qqZQRERh72tMsU305EbRE1A4TP\nVbO1oFxHxknwIusMvPfjnydfrRmMEWA0Fa4jEiyqMDdae59umuiVI2LDLTOO\n08vv2PXX/JmWCE1TFchqVVUc0W3ljncd5yZC4L0/uquW6u00GJ6hHqOeWuIW\nqTHCTwXbaqw3Ev5IVo/XY9EoFpvQ2HX/o9AWVmO9V4u6QbWSRE7VXTqLmfvm\nKL4S+6oDLKx7n7jL+dFvoF7LC7DocingX7anywICRlEisxcwgSglXSEdccZa\n/IeX1pacoPuQrjew3bDVnK6AVUjLQR0VZxmVc/j+hAD/q13s644VTw86Roiw\npZCFunhepFiCkLLOx2AyAZoqzMW1YWGDC84tmJfHGNbZFHkgyMLYx2tLGMVM\nHV/Iv+yLjQ/scilbayfGf8zHhcsKdOR9DyDizrHeEIEyB+zoH8EpDOFCAtNA\n1rrqG324VDZ6jfJ/1eHC1vERGdJsY5S0O3hi3rJA/N8HjfK1M2meLpghxQ6o\nOX6M\r\n=N4zY\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210427.c0be653.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210427.c0be653/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210427.c0be653","@microsoft/bf-orchestrator":"4.14.0-dev.20210427.c0be653"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210427.c0be653_1619508113798_0.48230874411101565","host":"s3://npm-registry-packages"}},"4.13.2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.2","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f31b654f06f6aef5c5dbdea75e24ef752ed7c8a0","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.2.tgz","fileCount":33,"integrity":"sha512-Y4QCHaAkrN2R4kzWC6Epy0aItPd6bIRrnhwPJZaU1ST9+UzX2joYUfEDD06MMkMdaVxQaCB9Ys6b9ieCT/yTVw==","signatures":[{"sig":"MEYCIQDZRDDfu97CR8eDK2Q2QD8YDQx1GKzmfRZbQ4pxoBzn7gIhALOg7UhBE/0PrtYyeIQmQKZk7ZVgLrJab/CBGuYcr1Uj","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123359,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgieTVCRA9TVsSAnZWagAA8pcQAKPGW+ffCF9kOhIlq6Wl\nCx8lF+N+WG8f1fRwXPlrcWpWnk3CXEk4YeU8LxsR8qtOKBfwN6qOFtPEUhOK\nYtMaw74kaUiilRtGA8KX4gGRbNySOJBHaECGWl2jpIGu3N0mdTWaVE8ktAet\n5CKYSajv8q+nKkCdq+Z09W4ZOerHi+7zF1TgDcMkkhfSR42QvJ0p9wdDcpaB\nPZW8MUdZAcgdbX8MoFcfWDg7XfRYbeJEZD4g4BdGmsOCFGOJDXMbbIxnxMUA\nOVe3tznlvAezvGNSJYS1oLl729vHdZW0NOhYA77vJ1lIISUnLIMV5eb9ycgF\n87L+qu9ZD8y5aa+Nn2JPfvetXCFvH6wPjXeB/hxlca3e6P1VoBskJy+Jrvxe\n/ZBmaJ3n2/rFeFl9NpB5BHQfMCaztS38Ro3/ofJ6C/ZFbjf6HX0LBKgFoAJr\nGB3b049PIWjnJA/YJqMrQaepj5G5frR/57Sqd9D6n0cRWvk/R12lrLDruhst\nPwrLNyhqrMeCiece4tmCma0/k8vviK5X/ersF/zdfFdn/izRehOrU0PUyyg2\nElZ2WMBRdaF/uoHAdW5Lt2LpfELfPFSp7vI93RUfQb2yE8bt7zFYj+ePsI2P\n/MExy6fMtkCzKjSYWewBbi17lLjCoA5uqcdEUtu7tvp77QlXiDcMDWryWwDY\nMB0x\r\n=hyAS\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.13.2/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.2","@microsoft/bf-orchestrator":"4.13.2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.2_1619649749267_0.12765734633736447","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210429.968ef75":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210429.968ef75","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210429.968ef75","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"cbdbd048272a01a2a89c1283432eac9737aca5d1","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210429.968ef75.tgz","fileCount":33,"integrity":"sha512-ClwN3CbgPiPpdlm8xlRDendvPIfWjiynGTdwL086lbBya4F7w12bM7jPRZVbeKkysJZv2oms/AJhu5TYqCtpVQ==","signatures":[{"sig":"MEYCIQCCeqd7iP4yM/6m3IQ5YE3l8oXTGFeTE1zbUOroAfw+AgIhANJ35uoTRG9qkasQhObky6v7mmCuV7CIaqd6l3hPO2rh","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgitbsCRA9TVsSAnZWagAAicEP/1Jr7etcFOn2scojmgQK\nH0lC/Pnwu+nKaQkt+8s1Ibj4yeTCZ5c4VwCYmbJ5LI4Lni0mchXQGV/SE2BS\nPMKCxfbduc8wlTjMu+kxBLFVwYQvru+5uveEezDmpNn4SzGbzpqD/X5FqrXM\nuXgDcnUTu5XUjwOxDNdyyJcl+0cqN3LLJypDEyvYa8uumq5F58VD0dIyvPV6\nYZns44p5IDvpRzxdheVwZdhL2T2Znk2y4CaxZOrHipJlCz7gPv1Iq1NTX1F8\nJv6Ugc8nYofQCrdJIHFzLpv5bmzegiBidaCqRdAaIziJyG7oVsac/P42vCjY\no/oEzObcGLMJZs+RXroiKiUUvn340+4jrf6wWZ5MtnFHzo6x1tetTzRvD/2u\nnCDMfSuZ2nqON/a6BmSJCq1sXdShxZENcACHoK5GvWM05eWUwrn+8UWM0C+A\n7sNTJIyqzVYZSB6kN/vWIQMRoqivV9Ccb0CUL8yX2Aysc+yt0spo41+QZ5MT\n4ngsHvLRXDipnN+dtJg6HuK1HbdcZ7f4yjv1dxWKH0sRWpaZ3PmNim0id/jI\nO3vNhauxEHhBuo+19BG7mPk2+Z5pMGha3YFhQRPcDpFB7QQVBFSAKn14J0Kl\neAvjKt3y+2EMe9QHwTpOJkiyiCgD+rh+zIxpQlNJCLgvksYXFNIC+DHMkraY\nmvSd\r\n=trGE\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210429.968ef75.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210429.968ef75/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210429.968ef75","@microsoft/bf-orchestrator":"4.14.0-dev.20210429.968ef75"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210429.968ef75_1619711724242_0.5795796101505024","host":"s3://npm-registry-packages"}},"4.13.3":{"name":"@microsoft/bf-orchestrator-cli","version":"4.13.3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.13.3","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"nikithauc","email":"nchettia@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f65d64923f7ea2a82a35334a75de39619464a03a","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.13.3.tgz","fileCount":33,"integrity":"sha512-h1QC6BCt2BCM31/BhrUKQX20C4F06n0l+UYBn+v4ax9O/Zds5A6hcZNojWQof24oSkS9K6/BZ4p9Dl9EZVALQg==","signatures":[{"sig":"MEQCICZNu5n4mpXjJZeGvIaF2YIEEZpdI3yQwfb+q2o9G/0hAiBQV9lgd6zKGn9bWHSAL7gI0iTz2d8HYeSlq1bY0Wen9g==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123359,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgixzNCRA9TVsSAnZWagAAqnwQAIcISx7N6NHS9cfwagkn\n7T+WiqYRwXtzjEbZXeWJgi+YicP1BXB7HlcEKlSnYTUe/X5WWje8PWhpKg6s\nm7ZhPxE0Zswkyxdu2Js95Z34YOTc1F1N6NnPXlQoPXKdjyP7ZcStG/vNVAYy\n9A2cw7P3y1scG2TSww86H6MtdG/n3VLyz4vQc8guGn4uQ2zQP6Tu6grhNVGJ\nB4KIxMNKeaeHpA4lbEj67rSb9NDEi67gddtiUQ5OX5PhABKrrhVtfiAdP21w\nlkv15N1KpPomuVSYjFhKYV1RsGJyZbA8k1i8Hbg/r1W8ZBNVD1+y6HNqkXBa\nptRaAZo5N9IvhT7OvoJ9H+LOu6DMMUct7TPeZAbMTlqF45QXBDPfC/HRApR6\nFoKKFDQqfDDKhuBR6fPWlM+oBhdfR31gKs/rTYP2gEqqTjI9WQuNqsO9Nooj\nNg5Ntz2trffdxTQhEc42OVyk3N6XcklAV4nuzy6pCpUNvOxoJmrmv9etLFqR\n5u3A2rRI75d5Bjzyq7v1MtsUHx+9qhEVRY7LBgXOPpZhICpaqJlldQxh10wm\nQFLzjW31jjbtSvcrSzjuvY1T7HfQn8n6hTIvLa1q9UUUCuU66SZkyyZPZBZ7\nVVafYtlXILqn8JLonqdLcRlAaHbi0BvYMgod+qjJ7EmP7tOTug1FP14H5wZA\ngRBE\r\n=ND/w\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.13.3.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.13.3","@microsoft/bf-orchestrator":"4.13.3"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.13.3_1619729613191_0.4929237725838371","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210430.968ef75":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210430.968ef75","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210430.968ef75","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"cb1dc00c9232f66b630dd992b2cecb3de947f604","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210430.968ef75.tgz","fileCount":33,"integrity":"sha512-viP/EMqRq2Tfua+26agJxnt1Ukw1Nk/em3m85Ca2M944YNjKq2x30GItVKGEQtRmi5o04EESByDa/XheK66a5Q==","signatures":[{"sig":"MEQCIHveCIcTTa1X8LLWV9yYFJJ0I85wQSjX7lVkLTNlc2EdAiAbP3C/xEnCjZKoJ/RhNv6TyfZ3zoaBJKa3MGijcqaJbg==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgi6+OCRA9TVsSAnZWagAAbjQQAKONDOXIc+StMXv9bcb3\nX/SOB5F4rS2atU/r1YAe8C/oyZdeZPT8sJ0Jaz+7XQgcbu8VUnkwXJ5KMIIH\nrAk3fHM2B397aHEXas+Dfm48CkGCGiWBpkJRTPIYBh9CxCQjk+uF2Kjmxv6a\nQ46OA3J7ulDbGqzegfHTXsWdUTalDem5hxR/8xibueqnNJmrdSzYqRiv0ymA\nBVEoA07jdihocahUmjaSNMDyxwpI/64JMw96XebgEFRzdbUblMbKIeXaH3DH\nbRvOdTNJpWrmv5X6GjVuEiRksir8N+Vol1w03M5V1bUeZRNtI5JyYRRlCXna\nb/N+0vsRFQHoAoFUHZCBqNwCC6YRMy7kTOtMLzG3AHQMZ0vwMUEQ80q0sY50\nNPJwLd7uTi/O3+0wVmmQm9TltIM2Ae9MFAlqlvEfZXCmVFfrgnDd2/wInOZU\nagkhNdNLV1/FOl4PqlMPKQj+3DLjKoeZgWzTYbGkQ8CRED9c0UzgubLxZcU3\nHLS5nCKFsd+5eDwvJqy8vTDkQs3ZnBjhp1PdLEtlp94iRYGAB4gtNZpQgi1k\npLNXOhqdqy8n+zL0uy2lbejEuTVvepHVkWpH1gsc1ASVz9rnUKkHzTievS6i\nGYtVvrYn1ADnCSE2dLT1tF7CpT89Sih/oWV6ptNTwd0F1IslEszsGZNX1qBq\n+nPU\r\n=JXew\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210430.968ef75.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210430.968ef75/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210430.968ef75","@microsoft/bf-orchestrator":"4.14.0-dev.20210430.968ef75"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210430.968ef75_1619767182194_0.04363305211812718","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210511.07bbd05":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210511.07bbd05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210511.07bbd05","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"8a7dd9872ca217f27594b43a0504f833a5eab6c3","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210511.07bbd05.tgz","fileCount":33,"integrity":"sha512-vFka8pxYAvpEbMppR9cyeZ57qsoEtiL+JBvxY7IhoXmbHj+TtILJRBl0F1kC+Qnx+mzmZ3XEFrD0BDFbFtlTzQ==","signatures":[{"sig":"MEQCIH2rEA3knkTrrF2zzcFYOvoMP8z4cMTzgdmPYbjczKE9AiAuZIN3fs/+XKzmLiw9CGD1dSosCCz9TbYYug/Byeimcw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123632,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgmjDMCRA9TVsSAnZWagAAhpIP/RUoeDnkETqu5gPD0iK7\n5d0TbHy+/qRpN9O6I50H0Z9G9aWCJd0PmEbRMMKqN86vPFiNA8KTUPLdev0b\n5qs2tvfONiuXFABGHTsbrxX/fj0A7C2KG5i+5yTDIga9Jbu2S3af5oNbkWYW\nPS7CvKJs6m48oxIhTheaLr6lJzQeSBuJR18ZSUoVcZL2gdVxrCqCtuRivx8E\ncKV5CdCVMJtyoj6gUtarjSgfw8btl63W2fs/HlEQyJQxG9RAfPAcfgRXpQ7f\nS2AfZXGieYou974AdnB105Fns6xHJWMq7A2baFq80RicauduE8+Hr2Yln67G\nW9MysV9Rz3ZEzgHExKNEhYwvkbQJ6CbitJvM+/dcpHW6wrOuSJ9sjKfmhMPy\nZkSNvCpg3iqySzYyEPY6j82xC2AIOeZIpOc1Rg54yI+PMisr8HC4Th/zl8xs\nIvEUI0wywUKNupU3gaA54tR4f5PzuJ0uVrLziRAeftgB6+Dqq5SltCAbFVEA\ncwjk0V6r7XKXfJqqyTbsuoVts9TQcI/o9ncQSRb8NdB4S7ib59kOc8ZPbSEJ\no1HTWEgMjt4AH3386bxgnYZ1gaVy35rEP8xcnDgqaTJB/6cmNaT6HzogUrvY\nkjxJR8GsiPOPRuSm8iMO/z/VIiIh3Px+kSz6YHFVZY2KpY5pQfF0TET40+z8\niBNF\r\n=IIeO\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210511.07bbd05.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210511.07bbd05/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210511.07bbd05","@microsoft/bf-orchestrator":"4.14.0-dev.20210511.07bbd05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210511.07bbd05_1620717772314_0.6878675974662394","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210517.4cdd4d9":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210517.4cdd4d9","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210517.4cdd4d9","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"9cf16949de97fe9c91e4945b2ab6cceb0b541407","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210517.4cdd4d9.tgz","fileCount":33,"integrity":"sha512-0zpyivbGaaPUgYxFM2ci/PWpkOy8sqPHfZ3rFsIDlrrZhSPcB7WohlMr2gGEYp8g8OlEjgsTFF3RPgxZPkxj6Q==","signatures":[{"sig":"MEUCIQCx9YG8rYNxLpOGjmyOfXr1BFuM6tq4bVuAEc89lOsirgIgTrEwS7Y2M4i0dwkmdcNr0XTg2wZGn5nqw1y8jCp0OQg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgohoHCRA9TVsSAnZWagAAYPAP/R9KuIPvUip5MtQLGnnS\nRCe+9qDMyyuGzC3kQpr/S6kJto1alk8oYlWZlwg2eYM27pDVJcrU8s21Owg2\nZEdCs0uqOcvA/OIH5wummY93UCF2H0T03Bp+e+IT7SaSH8+IZPPOZiUj9HHT\njTtoo78Mrgn3Bd8mNr+04PaN7sgxPHoTUpuBwznmcXfj1aoFXwzMSKGPhdD8\nBejQN1RYIYJYdfio7GxZOdHHs0zOwx3uDSv7k8oXMQev9P+mSQPEE8bBDB11\n/HaJ4xgpOTTHOKlyaAPx2J2tG5d9PYHcEJKG9IQa8DXemBWGpi8ArFwPz5SC\n+Zoo4hxBcPMFkc/hPnoZMCkCeIuwRpA+dvHutlosXxz+52/llOOmMlKlpPhk\nFZ8xb0Y9M7r/fv+x7XQJ/sMYjY1u9kGbfnng/h1VA1KJk3DSHwWKgeC7USHs\nQkCKy5khNpqK4CbvOlS/1pvpHBtJtcB9WCQl+SJzuQFKnotSpxGuMzCM3kkv\nrm3TaLycORZ2uBS2c/RzKgqrXa13fgN58VZ3zQhh2/tXsXhMsKPyFB8EQUec\nzmrsnyWtmt9NiD/iEhP8j0gE+ruNTXIrffAHpGQtGuDSJL5L6tznJ0Cpm7ZJ\nlx4gFwjLGynQ7qUea8W2IO5JfAfZu/A0LDsvYnf5lQkdRbrywlGvR8cQZ04Y\nNDhP\r\n=dz3K\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210517.4cdd4d9.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210517.4cdd4d9/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.12","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.16.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210517.4cdd4d9","@microsoft/bf-orchestrator":"4.14.0-dev.20210517.4cdd4d9"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210517.4cdd4d9_1621236231399_0.443339729584618","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210519.92dfb46":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210519.92dfb46","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210519.92dfb46","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"831d6b8748f2dad9b28ed4bf6e2e55413cd1f11f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210519.92dfb46.tgz","fileCount":33,"integrity":"sha512-uS7FuXmmPbxHKFglHgE5b4ZQ+50GjfMoB0cGCHH6njOwvKgCHcklsphjO4/V4YQ+m2C7jnasfHiHkHeaJ15nCw==","signatures":[{"sig":"MEUCIEEpeNtnGUwVEKEb0Jw2jNCS8RWFS1sHykAUXzUoZ/DKAiEAr7g9y6jvxKrA9lo46xFtajf7NfxiWJ32E3p3LK9T/G4=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgpLx0CRA9TVsSAnZWagAArjUQAJfqJX/NJ5PlVDZdvwoe\nXsXJuZIZVgfL1q6MvqVvRlVvBoHlFAyzhdoYIqj4XpoI4ComDOBd097HKXPf\nl5RBvUURcNRUxAWkKtTIxrWMOwfSS08MnQ6FcO4G5hqL7T+DTaacjeiQwdzN\nanrT+Lpn8b1rg272f42EkXrkWCipp4QCiRArzX03Na+0L1frOHL4ontc7NT8\n7gvQXhFLokogfJqSDKZdKqiCZCJFscL8xy67MH5TwHJH9G5oxmqNIhoVYk4d\nRDinUNSn7OXP1CySHzmtOnjwojARa0+NUast3szVZ0WnDRa/NgVKGonfiL6q\nk6wvqyUhm63W5tTuYf4TqZHzv7eptle9U4M83UEm4h9qFQ/UMSSpFtgRcJjY\n+nTiUCKBVqAzJNiaexN8ECptvVrW+EI9CPLDxNd9FYekgGLm1ZiQFUreZflU\nGHOoZS7OVJHbSTzxnm+Wq/Rl8CUBts0KERe5qWBJAgO3BEHEnULnT8JLsn32\nLoVzERl/lxCRJ96F9xaxSnPobM3MSJ9LxKViIYZZtrrMtdazTk1ZxcU/mhMc\neBSzNIiqRgv8GZnnPkIm/aQMvjNoWAVpIpW0YhZuXpgU3ETBh0gHm0l1+ChC\nvgYsLmmvVxzBuo1lUXSefamsquDBLntpauICfsa7CG0zQQvEb9A6iQbOo9rm\n4ZwA\r\n=ACb5\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210519.92dfb46.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210519.92dfb46/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210519.92dfb46","@microsoft/bf-orchestrator":"4.14.0-dev.20210519.92dfb46"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210519.92dfb46_1621408883814_0.5710609775764184","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210521.64e1d40":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210521.64e1d40","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210521.64e1d40","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"645cfd5bc5231aabbfb3fff9d7ee6755b484bb08","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210521.64e1d40.tgz","fileCount":33,"integrity":"sha512-Y65M1BG6upsKF80pSiYXfDIN3GG/8GUTuYiR7UVtnZsoSfPy7IOWZL9Ey4QG9X0nw0eeSY1MbB45MjgIVP+Gjg==","signatures":[{"sig":"MEYCIQC7OYY9F4rUthqMvp+H1A+bkRV3mF6elXJfjDW6JJBM2QIhAPDQYKgHxmshfja1gIXrwtxtYgRGhCAOP1PTQeQyZq0a","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgp1/NCRA9TVsSAnZWagAAOoAP/A6hXXN6KFdpcnxY/s/j\nDBTGqKTuCyq03o0ZC8b5LJ0Rx6j8MMEYwFt5k2CHJKfkcaD+U3LWcSzm+8Xh\n3khBMCI6jO32ZWtzgYDG/v3619JIAXHXejaTVitnA+Gajgc6jiYOgJNcyC6c\nDBLKR8E7FH/DodSKp6BgxjL7D7+wapzxKiN2HfhwDo9wb97zIuUdcZm0RMJh\n8yCF3dB/4CRNGsHPpK9A5d58wyp8oRPS9lzU91vM1Di1j3t9ONq3KFU1nws7\nHJPMOFwDRQB1S19dJ4/xl+RGu/AqDeWHHwBqpMG6Yh/uz/Q5A/gIL6Pg5qXY\nPQhS97ZGUsbXUytCXC1IGod3cOeVhJl+Cr6szm/OS+J+7swI54BbdwOahPub\noijSI6JarXeqTA+4TzvvhKlNtnO5fPLjng575q0qh4AQfDLPT6gTB6hFvFcA\n0CLHaoaKfZuSAu0r/0wSzFJI+mm9QsjO0psv1oGhFT/jyq23iYXdThbGcCjA\nf4Hkd3En8ez53F0xSnWKNwXncZ09etzPUUjD1npBo36jDxMml3r0DaoOQ0BG\nzh9N4Mri/BhtzEqMHP7hwqOgQchUoVcwDeWaQUzPjIR0auFa+0Ppu5lETroI\nvhfx+W/Lho632gOF/MX+c6TMWj2Aj3vXX5qhLW//RryE5/RD/H5bJuHQsEAf\nxyvN\r\n=wOm+\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210521.64e1d40.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210521.64e1d40/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210521.64e1d40","@microsoft/bf-orchestrator":"4.14.0-dev.20210521.64e1d40"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210521.64e1d40_1621581773457_0.016723951477218613","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210602.be805a8":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210602.be805a8","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210602.be805a8","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"b308459207861967446084a3e089a5afcbb02b39","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210602.be805a8.tgz","fileCount":33,"integrity":"sha512-gb95OtZYaaEOj+E+I1ieUVVJzRfIO6U9d+sGoAPaq4WpRQIKp0yRgm+RY2YSDag9NrBfLrOdChCM9vsskRFrYw==","signatures":[{"sig":"MEYCIQDkC6WdYsxsi2YEW4OBDr5nwLdzVRc48POmEGO0UuhHJgIhAJjtYZ5LVmzWmljaolM9HyAYDACW1BmJCgVUGiqgzRL8","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgtzKGCRA9TVsSAnZWagAA//cQAIn4Gw9OYIPVkkGnCLIb\nppd9TSP347YbfeNWgKNjZFdEj3WbMxuNTMPBrdyGyFhmj2b2eurjIyWhXTZg\n4EojEBX8r25P65/Zyp1AxuZoC4VtknRU4vyT4yVTDIEx+Ky3xPE2pqqHNlCQ\ngSjdZRYzBfONP/RtXkuBPjFSG9zZCIldVGOal0J7GrDq4aisNnjNQ3nXyQb8\ncl4AcwpX8tDHwXLy++CNr316pOxUti9N1oR800K/X08D4bfgIjCNemChn1Sl\niGgb5RgbZxk9dSU10CG/BDN/kWVcGqw/GQP+y2lAF3sfDFhY30P6coaMS/0D\nV9EPdCiFWHt4JUFHYUFKtE5KZj+E4hvN59Xjd24xb9fi67sCSBYgFVuGBjCB\nUe52Mgji+U5MTb/J+cKSd814LC6Bd0Dg6MuzRzp+dalOf5sXEz+k8g4kaeBW\ndTXMP3QQcxIiMxpATZO14UKuZsJB0mI9Fn7k1UqAesuarfL+NEYxIaSFbm6j\nMdry9m6Ccz+ftb2xsLqfJqDWdhn/labqKIqHZnLgKC39fkcR4v8qhN25pzSb\n+Qe5F4Oao1kyb/5kFIeMraN6uuCWWgs0MVcQqQ/wcIaClHbsEueAp3sp1gOZ\neENCX1eY1KWP+/71eyzXquPDbOy+YB2mi4dSjJvRgg2NS4SMXeuxA+RjrcBA\nt6CC\r\n=lNqs\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210602.be805a8.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210602.be805a8/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210602.be805a8","@microsoft/bf-orchestrator":"4.14.0-dev.20210602.be805a8"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210602.be805a8_1622618758344_0.4041118125840766","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210604.3f9ee15":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210604.3f9ee15","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210604.3f9ee15","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"08bbe7ad952c796f519d09e5fbc361ea09e3be15","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210604.3f9ee15.tgz","fileCount":33,"integrity":"sha512-L547kcPzOOQEocyLHYjtI90Sn88F/zuT2ANCHhSZe1624MxOtTF8DMyGEqLs5oq8xuWAOmmWSiOZw7iMCbx1Lw==","signatures":[{"sig":"MEUCIAj041jMu5D1fyL7m41jfil9/VYhfZyErXCc7mwJ5reCAiEAjKxTQIeIOJh8xBmjYvW2MzuOij6MoI/Cdn7zmDI5s2s=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJgudUDCRA9TVsSAnZWagAAQpwP/j5Bj6HpFNQpplybIM4L\n5QRhprnfYLgZ1ICx52bZtQK+d0gE0INo4Gn7wgtbbAI0ZY/1iDFLbgktL9e+\nclx5dD1g9MQ96jDCtq0i1bRCleyQojo9PP00xy0Dr8BIrcChDHrGgExDgCea\nrn6+FF/qqw5BBGZQ1/jY3KStzzob9EcT0KR+tAYmvOpOgsSlW0gzv66kIbRj\nuWHQjNCw00+6t2hjwC2oOczOLIf9J3dJ58FydBxieTKptPX7wcm74emnazgD\nP9dvbuoEv+qFpN7H1iMm3I/sRTw1E+ZUWnO5lCLjPcWbvbTG1SWifSN27jrz\nycGbHNvMrs4WMWy9Q4UzymqKcind3CjXvHuZj3JYrZaCBpd2WvAgmWbS8Lf0\nsSYq2hD9w8x9F8hI7lWIi41dbUzXonOsnGxUSjQ+P5cPN7Z5xg4wlGr/+tfQ\nQ4oia8LqP+5VJBQhymOiRTflagJEpxXQU1b/S02gzZRkJZhPM8Rh+D7X66CA\nmgK/uShdn4wzMCagu06aRWJEDpSFhK86ajdn9t3UHF9BY7BbFmhG0/wm1EFy\nfaEelQg1/98ZI54cTiQ58VQHA3+xzF48z4EPjlJ8yKZywFT8oPMPuWH+kqX1\nny9yKCUEwnCq3Y/SEyhKmgNGki4MQK0vaB+Po4VzTeqWo4Eqenb/tTRHenNo\nkM8S\r\n=+/wR\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210604.3f9ee15.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210604.3f9ee15/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210604.3f9ee15","@microsoft/bf-orchestrator":"4.14.0-dev.20210604.3f9ee15"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210604.3f9ee15_1622791427767_0.3831497128730126","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210623.37f59ec":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210623.37f59ec","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210623.37f59ec","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f0f803600ed9f344737f62c05e7fbff5f9576723","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210623.37f59ec.tgz","fileCount":33,"integrity":"sha512-HD0yzsqkWcbpaDVXpvCu6oUEvexvNTRUnk0DCG+tfN4KFTgIrOXm+Xcn8fR7ybA+UEFMO7HaBffXJo1yx604DQ==","signatures":[{"sig":"MEQCIE+lQy36GfRexiDvy2mhsPjykAMsCdPEjcvLY9fAnqdjAiBSatXgSQ8NiUmRoLPpXkTH8zRhjxHaz6nR+tdlfnvDcQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg0uNMCRA9TVsSAnZWagAAfNQP/05qNjzyYyKAHJe2D3QB\nAxp2dNHtmE6AT8fF17HWy+slVt21XYJQ7HW7/YwQHiobx7eE+dx8QVEnnfRa\nl8cpEak9nKq8c//Yh+Drpv8xZHeVW1lgmu1dCxGs2b/Pg6P5WgQqHy/VVR5L\n1rt94sm8CqEmLR3fLOWoLIQxM0vpVkjUbHLEhnsARwuzRQ6utEjEeWo3+U1r\nORjUY9AB5evV51knKuS6eCQmRevaI756HUHn5FX2hLoKhzzECxEL7mcnuaD5\n8QugU6mTKdNsDSA6ZFgsfq126T8EdrOrpFcLS/3uhupkIbmAP7Qk3QDKCgPY\nZpSKF4ezSrYBwdlvuvUxOjH0APvRBpXd3NQkT1+k4b5ACt8er5Dr81+YAmJo\n8pGrVwl7W1lMBI/0zaoZ16JUgrx0EndfJ0ArrgHcH5BSzXgiYF1Qi9ctPiyO\n5ZZ/+nIENooxwl+Grr2E33U5E42Zql4DB8rHBNW67Gh6+L8GCZ1NrRZptQuM\nM+AEKqFp06GqyDvrlwkmaz2Bc7SIOzr0j7u5vwumZYGitE8zU5Fc0wV8jV/q\n9KVMDA3C9EnOJ64sry7jEu4gknHWUdRUeDDq1IO8JeQACxwr9C6kL55aLc9X\n4uy2bQfzQOUHBOp2TPWmmwoZ2DinUjjfJU49YBzssSOFktXr7dcAJamHjhDr\n2eVO\r\n=auJv\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210623.37f59ec.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210623.37f59ec/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210623.37f59ec","@microsoft/bf-orchestrator":"4.14.0-dev.20210623.37f59ec"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210623.37f59ec_1624433483638_0.943294912486494","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210624.eac7a60":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210624.eac7a60","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210624.eac7a60","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"3527705fa6387e5cb255831ca920fe19da0c3f46","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210624.eac7a60.tgz","fileCount":33,"integrity":"sha512-gUSd5v0pcdlisF8EtlItJDgYIYhBZdkuxTIpQfhu4erz/QoVBJQmVDScGVFCjLSKGtHhYjBHtQFjklEeJSwWWA==","signatures":[{"sig":"MEUCIQD0d3hpABPTXUTNGWKnePFGSgaHzxv48RE6dSpXuNEuhAIgCtzQO7ADRSFnOnaiVrKguvwXaU30xCW6/BPNL1iOsZA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg1DLACRA9TVsSAnZWagAA7HQQAKPp2Ni7P+rYa30HNYk/\nwkOvoEZ/3JvctK0sOe7G86FogYEBKR98sZaQ+eeZT1S+3giYr/w+62TvbgoY\n213JU29oQRAogXDw/thXNBVY2m4QnxRWnxTDEDE2bMCRXn5VHNMhUbG49tO7\n550qukmbAfXWsEs0i4Kl/PUaQnP9uuGwuHkADSV4otAQTkl9k+M1i3xNofXD\na0A9X5YWVQNQ6zQu7ImiLw9CayEvcVazl7PpQjHidBp9cX2KFCF/wH+qCjU6\nLcw6Dr5dyQBKqrr5EVrZsAaLr/fnOZ4XsvGupqfcnyQAmeTavARaKDxYAe46\nbHEGhkWDod2tpPMDTZdRPbbMDScrHXw4lpqsPql93cO6FN38VsT5be+aY2nl\niRQRdbv+FTafLAyEkpcenh4+RPfoVh95sZNbKiiDkTGfScAo6u4qJgIn0gDY\ncEjBOJd0rWJ9Wn/tqJ6jmvB4w4ul03PY/mxzkkHG9xfdLVmQnOgkvjv8rE4H\nWTnVgMSOXzCZsG6amvCfu/PAPvW1sdAhYMmiVdRkkO8IPtLDo0w+E2kPQH8S\nllecd2kQlQS4wiKDsnde2fGkhvQXuorH1LCs2FR0kzk58lnE21nlqLpHvIec\nTnZ1YEEh0JPVuVINwtbw/+2Uz7OWVda4VdIqgmsmZyQLcRZwfwJsDGwPHgft\nDc0p\r\n=9m2u\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210624.eac7a60.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210624.eac7a60/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210624.eac7a60","@microsoft/bf-orchestrator":"4.14.0-dev.20210624.eac7a60"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210624.eac7a60_1624519360092_0.10939472853249144","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210625.132e0cd":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210625.132e0cd","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210625.132e0cd","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"b668f28eb900e91be85e7dbb59ad767bc0fe2a6b","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210625.132e0cd.tgz","fileCount":33,"integrity":"sha512-8O4sQ1p/Ste8ptKcTGEz9rSvhynjA1exmZFYPRQbj4hk3yb821Pz4J+ctG/myCfD3rslhIKYdVErPw8Jdsf5Jw==","signatures":[{"sig":"MEYCIQDiaQKzr2v0AEGojt6qAMtnIuAdFdWqlJKH9GLe/B3j8QIhAKCYoScL1XDILX/pbX/gyguBw07HIOck7hGl5/Q1W+99","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":123648,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg1YQLCRA9TVsSAnZWagAAZdwP/0sECCoaRp55wImSNppw\npMOWl0yccO+sJUO4quEbLVCbd25523LZ3OvzHYnQbU9ls7wMKPooozAyZSAe\nx/zq6YX9oDMFmuVZ9Q/dOIiL+5d+SmXUlvney7q6Lg2DiSjlYT1ryENgFYSM\ntVBrFfaxNToaTWLo+Ntucmfw11wKck9Dqw/4CLLxMGutsgi6kefGIKFpALsz\nH9YTm9QhzSxvgSvP6UP8y68H5CNDQwT3qAp+NPVv/pn2PjJ28vS8//1EBX7o\nhIu/Vr0XEwe0kylaox2veq5xAmGB1c0RmtZdqN7zNy6lG/LZiv4dx7HvGJ/l\n+3/e9FStJIWdFCIuoeVpYZyuaQH5+5gtzVgD99A+l3QDY+eIlw1NvrFfo5PN\nz9ASU1z8Bl5azykkrZJFYN9nEL+RmN2ssB5R5pD+NxBU8AHAHUcKdKG47FWI\nKkOQ2DbqRw/HT06i20kmQc9W2LKTCVO1nVslmA/WyQA2moe95j9n5aEg2LfW\n8nbcv0qY024Xf70m+90p6MY//hTc3fBKUfMh3gqW9qjRE9tgW15e5WljP0Ca\nKGVve/ZVo43oWfqRHl2YXeyEXwNDs2nHh+jma7S7ZsGyYrhPT8QpBwlmqeve\nMLiA0r7kre483/ceo6W8qjQP0VWNnphzh64kM42KjMUVMYXqCFS6bx8Gg69c\ndo09\r\n=5VgT\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210625.132e0cd.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210625.132e0cd/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210625.132e0cd","@microsoft/bf-orchestrator":"4.14.0-dev.20210625.132e0cd"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210625.132e0cd_1624605707186_0.5970597591291249","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210629.e97e6b0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210629.e97e6b0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210629.e97e6b0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"e232995eb6490e990b50d204ec1b147c7afc06da","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210629.e97e6b0.tgz","fileCount":36,"integrity":"sha512-QgaszWq1H8x3gwVgJjN1iZSAFWd290ZKGvXMkAnHAgFDJU9X1vV5cNq0o0oUmcdvn0mgbWN2e8XXx8U2bhKaeQ==","signatures":[{"sig":"MEYCIQCWcf0BE5Z4xb4TaLdEc9ASH487RhnYkxWsAu78enXEaQIhAI+qt0t1gwOClCUT+0m6CaPPmSXCDkSiMtN3gmG5193H","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg2sowCRA9TVsSAnZWagAA0nwP/1MwSszbtA7vkm9UHYv2\nJJgsgf7vLEsOy8F6yG5/uj2SzrGGjafK0UiC3m/nJo/Cp1WX8wlZ2dns9IQi\n3cLdrH9fLfTA+3EHi6y56rW+nUJA/taSVL42tK5XH5bWcVb4hZ7uNCCI+Mul\nW4KqJ2gwLaOQPghr+HEMF7ssXeh+O9yrpvWvQiByDnjIXhIMSgqc5o5YwszM\nJsWqnB/dMjrEn9W3jeami5A6idit/5nsOzaHNanjOCOuSOzEenkD2FvVjTbL\nsAJNwTVpuOLWCHVF8HJyJkN6iZHauoAOpYrDlwYqVvcsuhN/lbNr7mG94+IX\nNWGQ/SqhGpqfOOnlDykk8kqkKIyENv2AT/PHbXzP7Lj61DR6DrtkZ8mQW4Yd\nEaRxVUia8LILjgiLnN2UN7h0jjI6+lqUssrn0mOsoqqXgDWhoMXx6QPH2WEx\nZfmMe2996t41OY60V4i5LDEiBVMT3SH4XtiKbv6Bc0AuMreELtrh7WNXqqQX\n0+bh+H6nUWYY6cYzPKXinAazvCBOBLuyO1KKUBda3v+zELuR1PiZ99dorBm6\n71CzrHOwfViF/BklLStyhsnIQ++pHEymzCgLrUFWg0nYN72iEpSyNpLnqRSf\ncD9yRjY6kDatCoKyLvySiSOe2lIqZUT/24dvL+L1a3avJSGVg9f8ZGNnWlZF\nvKEs\r\n=PaB5\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210629.e97e6b0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210629.e97e6b0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210629.e97e6b0","@microsoft/bf-orchestrator":"4.14.0-dev.20210629.e97e6b0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210629.e97e6b0_1624951343582_0.9899880967430872","host":"s3://npm-registry-packages"}},"4.14.0-rc0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-rc0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"4bfaaa06ce426bf368e2a758dd020dcc8cc4224e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-rc0.tgz","fileCount":36,"integrity":"sha512-U+Yivrbe9JmylDGskx78kjjphrCTzm5rBeoEHh8W3uIkX9nyGzej0P0CL/M5nbTdx0HaNeI7+uZy3K1p6LSX3w==","signatures":[{"sig":"MEUCIQCUjj79b/5zm6jEL3FZ6HoYCvjDfzFmfRuGkybuehRArAIgHIX4/wxY3iazkFGyndWdkehnwQ7UOeP5o6woIepDy/c=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130633,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg22EmCRA9TVsSAnZWagAAkvgP/0gDi8YmrwYIjbada/pf\nSu2Rq4hOfdumGfYXwkRlQN2gvkQ2kdZkrhU4heVr0+JenxrWpFcDqDBbXHKX\nEyRljjP8eTXpbjimy20mKX61JWI09eIejMYGeWFZtvS4HW/zIDlREjWInzaY\nIcPmvBOMVbNTHUgh6BcIuNuU6aqZvaaO3iCUP93CatVilYkWDBYbN/KIbvn5\npaCX8yvgb7FXrhXqAMlVipPHoEGXCQkS8IWS1QGh5zQwq1Pxg0wnYGi2/DV+\nRwVlZzV/aseS/nWJuKoKHRLNDqRMSpkWv0MFjwKdPRTCAl7GO/K9UPliYuIz\n0d0UBQi67ij0n4si7VJDILHgXh+xHZL4fGU0cutAEtvG8Sd5IECHT6RgCima\n0T2nRxfWqwSGR6W1158sc9GBgWfLGHc0iVbGuHaSKySqyNMykCsSzUycHiGp\n0fcusC/xAd0/bqwblvWC5yhNli8N4KwzxlcIfreeoHNOBOJq+2cEo9/i8p6B\nKO60gq1ALNvsUmQssxqUtp0HLUtDVqTL8+iX0w3FOBIOnDxfhXaMP9Jnph7s\nvLsSve/cT1clc3Uktl9cZxmotpY5tdBQkFb9gKgyvW5fIWYTRhqQ4GkHXT5U\nJT/2MuuTFiqPK/VAsfeqQKSvEb3zu0HCdkhYEXZMR4kJk48hcu4vZmJsGZ0P\n7+1q\r\n=E4Cw\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.14.0-rc0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-rc0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-rc0","@microsoft/bf-orchestrator":"4.14.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-rc0_1624989989675_0.39840200259595937","host":"s3://npm-registry-packages"}},"4.14.0-dev.20210630.8078cc6":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0-dev.20210630.8078cc6","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0-dev.20210630.8078cc6","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"46c94ce86b44ea71d0c9eb34c326110c76964a71","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0-dev.20210630.8078cc6.tgz","fileCount":36,"integrity":"sha512-EBcL5Y6STGaFV6i1TjfU+UHugr33bNkjcAX9GjcNOnLeoQkdUgo/SuYguOGA0FsT3NUD82j325/ELJ3QSotxnA==","signatures":[{"sig":"MEYCIQCeVgIloBTMk3NAPHW8S0tw93+vZPQUEeDYkydO//pNyAIhALuo0yiuHJe+3xFrBWRVzXPljF9/HoT7TGgUcoA7PUYH","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3BuOCRA9TVsSAnZWagAAchQP/3Qch/tIlgfr5LgxzRch\nFk98mc1jFEYOMMlxehvusfuSEXL/BQ4lj1vmTcCpIUZ5pWBaNCp0T7zRmUqD\nI0zxoba/2+OHv92mdJ4l5vAF/Z+IIYaVwU9MHNhuFdrWwcQyFOyiKSVPcLqo\nwpaMVdTAltAS5HFGRYGuSmeH9RybPcjQI5psscvQw2hxL8+N7uI7Ovee3un9\nbQSCZ0TLmm4W7Enp7Y0iJwh9stFHRAQIGCRkKf5800cUUgQ1up98d4JP20VT\nS2UJkv01AVZ3b4HEK2QT7R3pVMo+nW53ighCfCA0xjYRzgGgP1qlsiWvslY4\nmq6NX26tX4HeRRaN2q29LA/jYKG8cqtL/MDkl/nF1b4L0pLU0WDNJc7AaS67\nbsDc3y/f/1PTPB2HG16fQM5fTLgfteocBtAK4x3jUJnLpvL87YBVazd9lsPF\n2D7QxO6pNgfSrs9V+u4Sq1SqkZ5vXhTp7Mf1RbNcpa9I18VWmc9e+fSfHlyG\nTTvS4VahQsXumcYsPyvUDS++s005TNKBTbFnG4BwF1fLGU/tV7kP2J3q/Kdc\nkx4pUnKpLMwYshwblx1ABSkEtaBooSD926J1pa59sQUXRa7QGhekuQ67Q3FJ\nQAXJN4MYu4tXHd5hp91Yws6He/XZfGwjXmQAKiHZLffDKtHdrCKPN2G2Tm70\n9HfJ\r\n=eZ2A\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.14.0-dev.20210630.8078cc6.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0-dev.20210630.8078cc6/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0-dev.20210630.8078cc6","@microsoft/bf-orchestrator":"4.14.0-dev.20210630.8078cc6"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0-dev.20210630.8078cc6_1625037710581_0.48053238445388624","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210701.be9cd6c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210701.be9cd6c","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210701.be9cd6c","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"e520e15094f16d59b0d55b593059d96f98c3085b","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210701.be9cd6c.tgz","fileCount":36,"integrity":"sha512-5wXmpojZYA/usFYX3Ze+q+S0i8xxnCzMcsAVHENo6lGzbPTCbIQfGfnZHZdDb9Y7zHwf2jCMY5VDGdr2X2AHFg==","signatures":[{"sig":"MEUCIQDhYe69Rz/9Ksu2YVAqAdCVoxzypE+GZtTvE0ECv7d/4QIgF3fXHXH/U8aQohGtM9+mqkksvQxrCdYtNpdzcbvu0DU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3WyfCRA9TVsSAnZWagAA3V0P+wRjqWCGaQmzdme39jso\ngeAtTyLBEOmxRTgP/Jk+ckCduGNu4fSQgtnG1rQloYYJT5U7gAPMIOW2K1ib\nvh0eLM55Ka2DIP8gBYCGAuJddDArv8K4bQaLNUHJO8ICSh0P8vYL0seJwUbu\nHHZMtBNj3W+cJSDFBrPlNLW9hWdNDKh8pKK3chtO9WoFCY04kV4OYnu8oGMt\nv/jr5G37fQkRupYccd8C7MiZjltgnjWLoqogFAjYqwcAEaqUmrUagxjGIETK\nLImtghAWwnAqj55kOPij7Fp90LuQ+hPPSRrGYA/6lM96espqqFIV60MFarzY\nYTMhWuc/MbQfJwo26O18tWpgy18otFrrOvhb9F+CyZcA9Gns2pLXWcBsJqCw\n5v8En7gEm6TdmQpwmXxbCHUxvdzmuuE3Ih2rka/PA0W1Q73Ol5ORkxzbd/g8\n80i/8ZllQD5AVzCnFjIOtHDujXHzHWcHHw8ur+NKrRpwOtil7l38Ttqhs2xW\nAzHV7vtRkkK6Gm5LuV1HjdHpRsyUaKf8RHw2IuKYxEu95zJs+/tYZpdZTuAG\n3850HHR1PDKIiDzxsC0830yQObPNFRNQQ7YCk9RBf2/rpn6sbU1OXMcYG2Pe\nu/ikG5LEjjblR3WReOv8EyERdEpUWcLFlEbzZeGNT7q5C70adAZl1KI/bl7Z\n5jAy\r\n=0zz0\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210701.be9cd6c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210701.be9cd6c/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210701.be9cd6c","@microsoft/bf-orchestrator":"4.15.0-dev.20210701.be9cd6c"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210701.be9cd6c_1625123999459_0.8671431040458177","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210702.cbf708d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210702.cbf708d","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210702.cbf708d","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f29f2c68f0f34847ea09ddbf2a109269029ea296","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210702.cbf708d.tgz","fileCount":36,"integrity":"sha512-L7NXPeD2o68b8jYq+iPb81P79+2FDmoQE0FNbM3wv6KfqUR3LZdgswGn+SASOpI18kecdKeCDjcPVI+XKF9SrA==","signatures":[{"sig":"MEUCIQCNhc/9Rjl0P7eK2WdoFv1xt5qKxzXa17rfoUQKh4BJkAIgAhpfSrOX86FubYNJQlFLuKJEC/BeEaWnDCK28qeouAY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg3r6NCRA9TVsSAnZWagAAlxAQAIla3E+nx2TLzKA4cWPn\n7CUqxMhlwGC5589RPg169ajRyjLUQgq4oZc7vbjWNxhwllwdTSBQF1mOlqof\n0JngtZD7F5H6vQCqzXp9mEMfpPKdb22fppCXcfX70daJ2Oeh45umFRdaYeJN\nyTaUdj1HLgmyO/+yG97SS9+iHD0n/VJtJN7tiD9KdBL4BDA2RWMfXTRftUtJ\ntzt6eoTtPcBEt7wLkNESWEhyDQkyKlX5eT/ySRtqiDr5q9ni+Z+yaFOuBfAM\nCW2uofYR73rMWA6hhV3m08CI/aMQwywr//Eqo8pgFt2Y9dVHPUgIJG1QyK45\n3DMRZmi4JgUffz9mD8bxCEgyGP1kGgM7Hr+LLowZGcP50dKisa7Bp8PDvOQ9\nPflsz4HSmTgARWHOvx9PQUV5A8ml3gRp7VQs+/aKANSFG2zpSjHFgOAqAoJY\nEdAwa07Q0D8IMfRCW19R92l6PDQ+H9owKt6D+zttoamKU7FfwPJf1pzqedYq\nAW86fksdYwoF2Cv7zMOZ9UJxh3ZaICFnWmGKIoDDYrQziJtQDRjU0AAnxqB+\nIQNG+KApFdW4s4+Iqgr2sxJA8ho7A4eJv1dojBSjOibRVDgQa+fcFyKVH2Fw\nA00cjR5/JfNwtXuucdCcz1vGMQj5foEPuGqzMJRIyRAhFPHIdYTvaBcB59eG\n6b3D\r\n=+RV5\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210702.cbf708d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210702.cbf708d/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210702.cbf708d","@microsoft/bf-orchestrator":"4.15.0-dev.20210702.cbf708d"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210702.cbf708d_1625210509436_0.21467352948525242","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210707.cf471cf":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210707.cf471cf","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210707.cf471cf","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"e4757077784d908bb05d4bbe62195e4747cfe2f9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210707.cf471cf.tgz","fileCount":36,"integrity":"sha512-qBesGfBdjP6rogXz6Zrl2ensc0+y2DoqyeB5/aDvIG7mFan4qG3tlg3wnsUTY2QE7KIGbGKZYb2ccMoHHqSfNg==","signatures":[{"sig":"MEYCIQChKZ66owxh/vsEVFYQAGAnUmkLCXUlFIUTD3ZxIyC+vgIhAKdET8vH2DNn3Wg75BAPaMrUSf13wA2+3d++s7VF/bbR","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5VXECRA9TVsSAnZWagAAxukP/2v3R3kVzNKsCrRjEAs4\nQMk98VXtSM4HbPXpXogjhWLQ0f9sJTuBrIHxD0hR+3CXA9acLj4xyYmagQjm\nGqc1fov/nP1Rfr2EI70DQKdBem1FDX4jmtkZZSed9FIB3RcuGALP6fgDgII8\nhTnXhxdMzsBy6nYBYJUvzXK0XFybAUA3ii5C2lELrAUROs/GqULqRW29FKaC\ntUNPU72Sa5NOjSBQ0dNZJHH1Kp9bAVwkyH02S9sJg+2sK6sva9xH+NqDfVVK\npvT68ptU4l4GBbxccLgK/CqNkjaYppcck15S/aRhbzkXS9JXvPwLYYuxCT5y\nGrpw35yhN2Kl17glYJdeRcCN6jYHpgxiOpC3chTE06ZHSWfGX7m0CC38eb8K\nb45r62v/kv0lV8WJtFsUI08+Id8tXdXhxODkx2OdX/m4ZWjwGpOSL2AnQrkH\nQRfKiM74WVhxzDLnoFENOzEnerKhVM6eTUBeFc8i1H7J0O8Q30i67RKKQcjG\nDG9QN/FwHT1Zg98DU73efJQ32TQ/6tLDGKX4kkwLgLHbfj7Y6/Idp74TYBEN\nL7QNiGnSAbpgca2k9ZZqsP1MxFrLVCjLsnI7HcvSYGOXRPHns12wa6MBh2P2\n7MxgbzO/4Gp7LZ6/Xu+nFtq+5fcf3UZ2l9ICeFEb2PZuxWF7qc21YmdOyR39\nOyOg\r\n=a3LH\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210707.cf471cf.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210707.cf471cf/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210707.cf471cf","@microsoft/bf-orchestrator":"4.15.0-dev.20210707.cf471cf"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210707.cf471cf_1625642435675_0.32743484538377743","host":"s3://npm-registry-packages"}},"4.14.0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"12617f8c6f571c2d22a5fc9bb7b7adf5e5bc61b1","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.0.tgz","fileCount":36,"integrity":"sha512-QJxF0Mib8smZLMjX+LnhY8cTd+EN/WQZpHN34q4oiB6vc6Nu8bSO/HoJILT9dhBvShbRu0cWFnPoxxojM7xyrg==","signatures":[{"sig":"MEQCIDVSvSML0EvaF3AWSV/k8Tto6l/UgjD4pKsJ8zSMKru/AiAOMQRXFvxsbxjWhuWsrXTYDbB5pzDRNG13r8HTjMIpUw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130577,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5gAbCRA9TVsSAnZWagAAxF4P/3tTvV1OZxF4lp4rHCo5\nn8wyp5iKlhQSbuEGRAEXk3/X4X3d7uVZHWMSuASqDNpHxQmRRe5bT/FJSRGa\n+yWiEMldukzyMWdU/ioYAWj/DJCYdgxgjofsKVv7UHAoj3E8qN4OKWmSeMVd\n9CU29kBw9cctHETmLKkVVmCpasTI5bMtCxT0nX/eAuemTf5lqZg2N4Adwg6K\n5VGVbhMDE1eyeC3PfNmXokxykmuG31pAweYgZtu4sGEfzIBhHNNbEmbqu/iO\ntfAErJtm7V8pKhqubFOkzbde4RBNVghKpn/03B+Z+pVo+OcSAm6A6eEstbx0\n24Dluzl1ldDX64x4jqS0oigavxAWkscL+vpji2wHLRGK56NrzfEPsQi6gFbc\nRXAHvYgE2eB9UCttEt+g8kjufZfdYVHqUzZIUZUwEoIYnVo4V8lx4hUTHXv5\nqDHrbF78/YfkIvNYttgb8xgaXek7Y92ng2fxLx70CxqgIaTOxEhgxrbAraU9\nlZPCTYlGDt2o10glrIsYu7wOmu0DUgwkVtQE5G01m1FzNaJZz/PtjWhl1tgl\nL45HNSQGUlEwbE7dtJrXlIxB4HqSE+5UqBtly/RsjRB9Em59tzaZgf1CTplg\ntsldjMZip4tnW8tkwpJqgUwFZOOclIIRNaQQ4rG+eTSvExCPU8N6dH94x1Iq\nz3LU\r\n=AjiY\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.14.0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.14.0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.0","@microsoft/bf-orchestrator":"4.14.0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.0_1625686042886_0.7124805859979984","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210708.f16ca8f":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210708.f16ca8f","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210708.f16ca8f","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"6e1109b4134812cf5a26fa0aa54a6127751c9d92","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210708.f16ca8f.tgz","fileCount":36,"integrity":"sha512-k4w5NBrdkzegy/JlgU5TGDPG/gwSnQMcb26xNsuxrIkqSPiZlE+Q6ZKg4wC7JLBz+EPchf1qhua7Wl48jlGu8Q==","signatures":[{"sig":"MEUCIHAL24zfmp+igy5X0a20ry6D76YZtKvdJV4fnFwBgv6HAiEAkxWaDUaqAjzEy3RrwCL2oc5WaB6vthaFyPYVaIGqhV0=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130015,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg5qcTCRA9TVsSAnZWagAA8JsP/3LNqIK4/j61VtjBr11y\nqPknbUM/9SVkamHv+uhwXaFf5k2aUd7MnSn1tWXYWO1HWHYZzj8f389jGrno\nMARtwyQ5YtmWO/vDsTjHYoIkJIHiCYDQo69jP7ycUtoCoSrQea8QQgMhkYhP\ny7yxJBWidvt+hFCkIb27MjNXVfrfXWTTb6uhJRzCYLQKrwBJIHFcvrx0YQ9O\nXdYkCktWzyTR9vMSpV1sXHo3kDQdmCmZPLiReozzEy+htDyLY5CsLWz8hWNP\nloamXDlxWWjFKRd/AvBsMirr2VxKJXZmb0a8RE72zX3XRMd42nrS37k6MI1s\nB69G1/5FXQy+cu0EAzktz2WP1bLIObgUFOe8oF/wHrsMt9v54mNFPAurZzWz\nggWVKomMI1PCWA6UdeRitU60nPoMoHoa2g0wAbHMIPjphNvZ6lrzmvBLIo7y\ntndhjd4we+njaPBzM8triqvILvc1QdJgAkoW+BNntjRLcUGJRgPtcj5Er2Ss\nvzTL0EN9zRNX2CKNRHAxIKyPLGlKeFJMRyOsD7zOzXFuzGe/bGBgJcCwINpc\nQNOUjm2WEvJI7qiZmFw70kPV6+FfcsAXSQi5ODOEZM0PbY2EFV+Spx5aUhQu\n3u1LCaUl2/ZI9h3tpCWNaN+IJbsbFbEwtJxFXSotMcHMMcSa7PzJZNM2T5xf\nXb1t\r\n=sz2T\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210708.f16ca8f.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210708.f16ca8f/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingname l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingname q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210708.f16ca8f","@microsoft/bf-orchestrator":"4.15.0-dev.20210708.f16ca8f"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210708.f16ca8f_1625728787419_0.010606604336893533","host":"s3://npm-registry-packages"}},"4.14.1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.14.1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.14.1","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f5f99b2a37745622d562481dcada842324130904","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.14.1.tgz","fileCount":36,"integrity":"sha512-5r4Kmnw1S+jt7zBqkuNjfDtXL7NF7LhAEYEpSKl12ubFaz8hTRTngCbM73s7B13eBdToxJFq3cW9tj41rhlrJA==","signatures":[{"sig":"MEQCIFzvoCUfsTwAr2mZLcc3WJfHdqMHvTf6nAokvAbZE3itAiBnCBnEwdqdUelde7DelISWd5u2d3SUQ1iqManRhi1lGA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130577,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg50a8CRA9TVsSAnZWagAATwYP/0MVZyp0Mr7Tp7azEPb4\nmXyijeURygL8/s3bMkwzA7DtD1ZT0ZGZY/MQG9eqUowzYjth9WTsIeLsQj+j\nWolzQzUDTLkDCp+9BmmzIysOWwhc8mtoafn2lLNQeQugV8zGNguoJ8sxbX2C\nwzj6JhEXL6OuKpVitjExWS1mTP9fJSMkzw/TVpjt3z1W2UPG8MSJmvXx7QLI\nqqF4bM2NOPzlhleYdogTAGGKoN4+avVkuoGx2eJXGlRlTzYm4D20VUwr0iVq\nh7fqYEjPFD87nURj2r9/sO4qnZPuYjjmFUnoLqP2fLjJC4sot9FZhogt1g8l\n8nD4qvOZpRytJdlnYiTDtTL0KAbdVOGQoR67kbZ7UyxROS0zML5ooNAOJF8k\nnUa/NRSHJX2GypPRmJPVJ4XLS9CMwBxMVBf7Zjj+cFXOm2rJHmsfe29c9JID\nz60cUSsxsZjLrsj8FuT8Ii4vjDN90jKZH5wvC3LJtnodf2IQic0njO0Y6QqY\ngvVd1MsFb3dfOrD3qndv5bPk/pfVq0bOxsVDrAW0RMBWyC3o7VPAel3Tl+2w\nkUcoxHOVf6XAF0lf2a6pX4Ixsa+oGcTQLTJqAkSu2GQv4lZRNQhUBKQK3sJI\ncVBuyKwgL8pTZCrDH871+qREhxyHxj2zjiGSCZ6Yb/n4aGQctMSE7kUF3+ZU\nhMN/\r\n=ntaE\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.14.1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.14.1","@microsoft/bf-orchestrator":"4.14.1"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.14.1_1625769660328_0.08609401300086073","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210712.eb38bc0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210712.eb38bc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210712.eb38bc0","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"d49a04bd34248e0a393f91ce6abb6e1f98d4c2d0","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210712.eb38bc0.tgz","fileCount":36,"integrity":"sha512-Saj6QHX9rMu05M65ANbM7VVrXNa6NT6kRoRWdnx3wMBXV6Z15/xUVN5AGFxLyvoxyUHe0Twft662FmZj+6Qb5A==","signatures":[{"sig":"MEUCIQDXsEFAtXOAbugE8u8LWdViQ4xJeTMTHy6eq7tWWhTxBAIgZ4fNi2VS3Ps4iaN57ShfiDqgKCWNgMrLhWtHjN8MatQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130871,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg6+0OCRA9TVsSAnZWagAAeYsP/0ItBxQ1udQoGZGvYrC6\n2PQV/5vYfkpgEeQH+UpetTnn7esfXNkwhlM2VRsQhb9l09UulGSyd/anP3NI\n7cWSE3/yZ8ryR3oJFJ2T4qV/6AD61kMRa3byMmz1xOH8pSeNDUPCgrZL+F7C\nIuwva9Fk6nL8/sMwwdBB2AWcTKk3HIzuklayd6w2sBVwFakCX7h0t8utwvrF\nKh0K1k4bQa0KgtjsZsVWmpEn2vJpVxkslxHnnlDYAJ0hfCMcw2rXSKgheOGt\nSYiAj7KmIbDdCLmVG/TusOCMpveXdCEfd0Z/+wBswNSAqNfBmklWB7IDCtG0\n5Nl65eS+/tAzRpbMAvkD3Xns9bUNne4D6jsxB2aHhinyohaCcVPuMjwvy5xG\nPDS8IRDhbU2dbMimphMf9o6ZOWo2G1HJo0AD7zIs3OEIME0z18gq4FMFY+Dw\nFUhYr5Yev3iITkXepK8shOZ9b/JTqjXtYHSf7/Wb1AoxeD8VMoMQLC6zdeog\nsc2cfSJqg1Bn3DqFmRHc+nhf47iVcq/qyknp8NTDuQYCYpD+0koSIDOXOLYL\np8MOIp7V4RboAbYz7c5f2RsEr1QRI16KnV7FfpZCGgTXqc6yQ4ZDCU3F/o8P\nrGMA9Swzw/MH0D89iwJrn8JMX16yM45eG89j35Zvpli9ODGKItsQYTNcnnMt\nw6/a\r\n=Kfbz\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210712.eb38bc0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210712.eb38bc0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210712.eb38bc0","@microsoft/bf-orchestrator":"4.15.0-dev.20210712.eb38bc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210712.eb38bc0_1626074381971_0.5319878620576641","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210715.ead1751":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210715.ead1751","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210715.ead1751","maintainers":[{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"botframework","email":"botframework@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"7f94ede91e304033ca69a413fd932a10b622a4b6","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210715.ead1751.tgz","fileCount":36,"integrity":"sha512-xUSq5nWbUhEGLWbVkiH91BFAmNd1f+yBoxNbf5ms1bJyFGcnQ9LXTothOlPg2D2kXvk3I+1VJGtLzBT8x2Z2OQ==","signatures":[{"sig":"MEYCIQDMKRmpmI7yLwvRPo4wZiptve8rO0jR+33lQuFYZtY1mwIhAI1PFE2uo8JesCsWExLUl7VNAX+Y1Cbw2Ldd4W07Oh1G","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130871,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg7+GSCRA9TVsSAnZWagAAWkUP/RHi1ihE2gy5gnBfTzQW\n3YwjK/vybLKOytmvvoZyiG9arGZD7o64dw87U+hByFmhEvruESjtAUEZCaNX\nS2Q9/aRXvGzBTX1BkrSZNNLFCba2kdUpv4aitj1nXMoUOnVicLxlaYiADrwH\n2ya/eDE7AEz7/Rumof93Zl9or6UknRbJtQAY5EZlvDfS2e09vVpJLddZ/6bG\n5tC2Qw0MgJakN6ansa+jvN3Gv4kiqWXpsBBJYbz9/tj2s5DpZJLjRFVQpHxz\n+KVj8HlLgGPvyKt5DgdDZL+f1xLKK/7NzEzSfitjQ7+7BaR2f0rW3MR6ifse\nLDM3HyAU3GCouDPMLr5Zw7Q0jAGUZ3doAq0c2mdSoX25JRO8792FaXwH425l\npolL4G+7sGA/KraYFwI4mbzxzdRxvWSeKbvZiYJBPaKb82t6pM8bae+LuIT0\nBTTXRupmKehY4fHw41bAJ/o8H8C3UKRic9HJug50mGA3RecTXh35a3QZPY2l\nUgPecbuanTXemIhJ3bIfF5qa0UkjD8TqmXgLPCVIRk0Xsp55dVrSHq+faJod\ni1xr6irXywhFQAykAKxKQkRmf2WraMq8desEIp6BvcGNWGM3/srYORJx1RU2\nR0PYRjGmnElomWr6ps4kt2SoLdQ3DWxLWNe6oHPxlzKmfW9RMYJtwXnnSuUO\nbc/+\r\n=+Z1r\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210715.ead1751.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210715.ead1751/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210715.ead1751","@microsoft/bf-orchestrator":"4.15.0-dev.20210715.ead1751"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210715.ead1751_1626333585441_0.4962147407981008","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210723.bf1f054":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210723.bf1f054","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210723.bf1f054","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"2738619bb479d79bc4d1ad495a4d49b7c618673b","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210723.bf1f054.tgz","fileCount":36,"integrity":"sha512-+ldUmyQ6fqZ16aZOn0AuE1AFnIWj9uko7ICs2xyHzSWOm7R65QM7/wHDPwo+64d1tSXiJG/ffxmNs0a/mqZupQ==","signatures":[{"sig":"MEUCICLlLK2rR+YF3rR5cz48UJhfCCk/qg4QMfcwT+Ur6gTSAiEA26SmZwVus4WpO5EvlVS35RrbsQ2F/aMJbZWlEd+0TPk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130871,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJg+m6XCRA9TVsSAnZWagAAQ2kQAJp6uwGLYSt7n3J/73lV\nu+gb6/fCj9dW3xtZj4HPETJe6PGgLFQruER3gqi+R3DkjsGc6lWgk+omCfTL\nSzD4GNAC/AWMkvoBWyjRTEE3STph35kzVY29rNcCsV2gKKPFQQ6N4+0sccaF\nz3+k/ndf18Ru6WXF9QZPpEJbGSrCfQEPKzMgu7D9Av9YzS2U1UX09kidWT9t\navNO2D4WtSYMn5HSMmQ8oU6H9UU5mTWWi51IVMiVEyZT9/AkOyd89jIzU9aQ\ne3H7JrAfvfNKcrtY2MgqNm/VpzXVwl6w2HiEfWU6rfMxkjw+PBu77nUgY48o\nv91JcmyD9NFjRJELTxSdaJ/1ApfY86PKrx58LO/nkIshjuSXvWvcN/9GmZg4\nUDuBFSM2YKrW5VqCINWsDCvelrmS9cM2c25dpbmdFAI1uHY+cKt+idern+l/\nlHi+KCooZ9USElWzaAEIA4rQljuYmdhY1MMKTiO2GhDAN18HRv5x0YmXocuP\nsFmuN1Df6Kz2os+Vpaz4taQ0vad9G9pc9YLEALAcHP/OIm4CwNZ1Pt4I8b9V\ncHk/neCkZRP0nactnavyj6rB5YxJ8SyXcBz6kudy7gAhu0sZfJUGhrolzPVE\n1S3d//yapOdNE0lOIKPePBJ4QtSbP44Vkm/uxLyCO0Y/JJIqBB0TGnBo4ST7\nfPS8\r\n=8SDf\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210723.bf1f054.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210723.bf1f054/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.3","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210723.bf1f054","@microsoft/bf-orchestrator":"4.15.0-dev.20210723.bf1f054"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210723.bf1f054_1627025047500_0.03753380600310163","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210803.40549ca":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210803.40549ca","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210803.40549ca","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5d5b7d110f9752e0c99c1548a9e3dc749f090752","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210803.40549ca.tgz","fileCount":36,"integrity":"sha512-mc2xJCkujErLKcaImlNTatnkmgC4GYIx7a9YBVy52Q5o8dN0r4sm1SWLT/rviDGkJrhmNOMxZpGfGu5IgmQmiw==","signatures":[{"sig":"MEUCIQD0lkM6E/4xBzbWuRqw9y9dJCqy7+NOfKnIvBLDI3Hy3AIgQeGakN3B5PyvUuWvIvN5y23dLW6uTi2N0Gu87NMUtVk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130871,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhCO6lCRA9TVsSAnZWagAA0xMQAKM2g/WBA4J1C+vcMLYH\nQgSm4Gw+IBF2dMjsCx51+kGXr7EyoojxFrsHbpIsXPY3p4SZzO2+WJ/gbiBw\nDS1gIltA0xASGc0NP3mDbtYOFCEW8En5zrR6kx+khTtr3yQsTLRcBa4GGI6O\nLbzPFzUxvueJ1XCPM3tuE9iJZt/t+LGIDUlsIb8is96gi0Cad+K01L5waioU\nS0OqqLG3eKo3bnf6tk+Kq9y/7NJiwVx1LOTY++03FJIEfLgd++1AclsDmkFh\nIPvdJ7iof051Mov7JEHph5rGjneI4FJXklDu++grIwgDIIz/fjjCkgswRFej\nfNgfVNhsROCsH6IVGjt43Nwwg9YjA6o5HKaAIPkzHNfKncgVJsMQDj7fSA/i\nm9iZfj6g102ITcRQkFmbX6v3WBR4u5Vy4zmeKoMhCc4f2TyMX3QIxzaAL90K\nb3wY2UKMHXyGHD2n076etEdN1J/BEhAtkV7MZ3vnrnfoytaLC16uR/Ba1dV4\nPnNKV9uu8LwAh9XfV4NgxSfZygXh1e9U/L5FFG1apgjNAmFK7HcCanRjUhtB\n6mOe0d7I2sZXRKWlLGOjQVv+ljLJufNjhHPMLsT4nkovMZbC/urrVcSt7+by\ntFbc4u3ZBzHpL1nhGCzZitaYLJWuDC9yi+Cu4n/mKX9wuDOpzvuRDbQ4hAGp\n35H4\r\n=f9Vk\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210803.40549ca.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210803.40549ca/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.13","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.3","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210803.40549ca","@microsoft/bf-orchestrator":"4.15.0-dev.20210803.40549ca"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210803.40549ca_1627975333430_0.9541872330860703","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210805.6321dcc":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210805.6321dcc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210805.6321dcc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"155a4ab2cd7e32ffae7e657577aea1aa2fa7a3d1","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210805.6321dcc.tgz","fileCount":36,"integrity":"sha512-FseFJomxCMZZER5BnZ4aG4bb+icp3ZVjng874HAbJLSUUWo1kOVO+/eBNeML7qSq6CUso8/3hsBnG+DyeNH5bA==","signatures":[{"sig":"MEYCIQDc/maIP4Xq/M/FkYwUf1u5SPShY5OGxY9OM9IVCk+ltAIhALdHttHH6e7c5cZ0Jxhkzko5dIQqDRTuSaXa35YtRqPR","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130871,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhC5FJCRA9TVsSAnZWagAAwiYP+wXgFV+m4i+hyeiwj+bq\nm+b6LOWVjjsY5J8GWy2R7yrt5nmYATomFa1SQQHpfTi/PB6NLo5zMH0hzQ7K\nfNvPge6UzMp0x0c5rPWzHdF10jrCtf3MkRKM2v2CDWQIn1GGBaAtsxtC7DUH\nS/H28DT/LbtsFBxOKyB3eoIvj4vDgVjT1sePdtpVye5bTjiwGUehn69lqlOV\nd+GtC6WcloOAgpW6OIn4NDEWQ59oY3kc244chv9pX7NybAHpbILZXMc0cSQI\n2Jdprg659ryKKcD3/0FiYOkVtxJfqyWl/96SMb7YQ8jbHh59LuTWrhgwIjhr\nhvSkSfNQTf9ouYQtZNjhFoKJtr+6v5O84xJ+/Q0tnXbz7Cu4Y21rXgvternk\ndIxhDm1LFBQyTF+arP6YDxFgYG/Bu2sfmzhshTejFsEVoJYIDUe2EBEFeObg\ndPw+4useDvo+GIMrmqTBAQpQWWcgFsUdjQ+6Vg8RIZlj8lrSRjX7ppq1da3D\niFBMbDIdedDbmGOqerg1epSNXPIc2GjpLOsu3swF9yZ4aP8HHYUD7dS0Ajjw\nQfqWG2XD9Mdcf8n3yPt8T34hezfhffMtZ/IKzYCOFSJBUPMiwpVmuZa5ZWl8\nPJy4KM1q4nyGvRRl+2NjRJe10v1NKvtEODRe/EPicu8+l2yZB5s0PljeBFD9\nsqRs\r\n=cpDc\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210805.6321dcc.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210805.6321dcc/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":">=8.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.14","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.4","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210805.6321dcc","@microsoft/bf-orchestrator":"4.15.0-dev.20210805.6321dcc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210805.6321dcc_1628148041591_0.8846716263378571","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210914.7a936c1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210914.7a936c1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210914.7a936c1","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5b5cb718cef84ea9c60eec2593dfea7cb84bc404","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210914.7a936c1.tgz","fileCount":36,"integrity":"sha512-gmeNaicb8VorcOk86J1d/h5j7z56xo3FmbVRlBvSpID7nLk+igTRv/RrareKZRtYHIuuQ26do4aPTvJIBGbuRg==","signatures":[{"sig":"MEYCIQClpf2aMg/0ruT7VIlF89qwK+C++AvXQ7Zc8elkQanyAQIhANBRv4gvVFZ0AB+KJ1T5ia0wSzAHNj0NOJMg+8BI79Bw","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhQE1zCRA9TVsSAnZWagAAK5wP/Rh/21FZJzICdc/U2EXb\n1pBCSGzjBH007WlJgrSBptJb4Zvt9VXB/Qi7c1+LX1uJD15+0beDWLHJhTPc\nKYyFO1oe02VhocWyK8Kxq27pu4yTDahh0vEkvpwx3a1CDloq189nMQDlCTUF\nD6ZEBdyOrldCpBxVMxkyc4e0AOVvf9xrtj2IHULlhUwkksU5fUFLg8r0hatX\n3eTlGZBwLrMVuQ/xYBzvOwTuV+JVY5T644AzuN++jVitQhKevfUfVYtnOjDY\n1g//eoKZVkzhrbW1lTQiC1vJI6ydbw1OX771wYPHYu8OmveNzmhvvgUzqHQj\ng8ZAnSUyDOGij4aQta2hts9gJTFNaUckH0y8MOO9e7EiZm4HIPApk2WigXv6\nQb23PjwV/GNes/Gm2VWbda7sLSoFq16AurvrE366C4/VBzz7TtQDM58JQ+3b\nqRQ/jbmrcGI3XMOtjBcXi1ymYO1rlpFxkUT5WH4ojK8IBbJYLj7tS7OiSSPt\nA0zV8n/YA/UG9WsbWvk9b/s5jjR2YRBfp/6Ig0MxN+IlYdVx5OgXnxJFRZsR\nI8umExlzo55T8SKh6QG/THGaAUD11DwFpC3AQ6IRWBQ56pifPj8xq+mYbIdL\nIc5bIxGD8/LKG+XVkPhFPr8FtcBi8EyGmS6vMwsOTQ1kVaRg3ZYXj4ujhmE4\nmLhr\r\n=hcsk\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210914.7a936c1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210914.7a936c1/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210914.7a936c1","@microsoft/bf-orchestrator":"4.15.0-dev.20210914.7a936c1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210914.7a936c1_1631604083289_0.8891619604511631","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210917.b688652":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210917.b688652","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210917.b688652","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"ced0f65d1388d5510d74dbbe0cf007eabe78e8a1","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210917.b688652.tgz","fileCount":36,"integrity":"sha512-HVyPT1gOhUcT07piJJedm9c+UGnHSNRXzd2s6Pp76FzHqRXypvd5Dha5Nb1/KgxQZfSvFuuR9mBI+6NW7xJSzA==","signatures":[{"sig":"MEQCICHyjpAoNMMFyxSAVAN8HRqyMSFw04cRwyMAy4yGUF8pAiA8aPgDt1x4ANR2WJpktzMufrD63pdT70+2QB9l06RQCA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhREKVCRA9TVsSAnZWagAAS+wP/iPuzNzAZideodYvSIVR\nXWyFXUZpIlEVALyj4WwPlViLD/POk4C4sCfuZdueSgulnl2LrPMEOukHFaPQ\nffIeeph0OCL/89xLwI+WxEE6vhExviU9saZV6hpkrhGX15mgjzskvpJl+oJZ\nRa25HHMGWYke36ZZN3ZytbETYaDKbsVz4XdEJN0tFYKC87nQkSBsB/gBsdy/\nGsTmKpSGijaxIJUcVQ3x8B09nDbItXMWUByhGKJzavc4jYWm+ZYEoWeagjRG\n9Tk4IU5dR7oQksSJH5oUkdn6ATY2FB+DqaS+2bo0WqGRzNrKcRyV/jwjDYEU\nlT4XKKlHq/IPEsFdnhoBvK07XgZZZZs3MtBveTBH2SaP7XLCjrXaYM26+yCb\n7EO+WqNMWKpf+U/uZxAjFIFSf15EtTkIiAWZBtbH/b38CuEG31QhOnVM2QkK\nPEDafBD4XVJqkAZOmvQakSKhHEeJ4NfWOJ7iY8N6csW3bw9YBoEBZMEC8x0K\nkWZ1GEX1pC1F3h6C9pBQ5mR1RlzvQ3IR/FDLHMOB2mG9d2cDhBM2FlR3S1gJ\nqyqa4WDQuotkY+4/MtFSaGSsZvTJgp6ZZUVI+7+76MPCstwv+KsMSN0F6nlB\nNV4hqA8vx6EijMm6IRQ4lNaGDeqndMA49gCNwdjiXpdFSQYglMm7zCz9Qv5T\nrk0u\r\n=sTkG\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210917.b688652.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210917.b688652/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210917.b688652","@microsoft/bf-orchestrator":"4.15.0-dev.20210917.b688652"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210917.b688652_1631863445281_0.48764665353764514","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210920.e4d38fc":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210920.e4d38fc","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210920.e4d38fc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"78b712d08f625b6a32a208b7e8a3311ad125ecb7","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210920.e4d38fc.tgz","fileCount":36,"integrity":"sha512-O5blSvtutN0rtjAlFzPrGCRWyjvP+k+rtbGfXWDIffNQqnyzGP0Gojvb4DowTD6zPhb3DmswpgYxxLVRjjrcZw==","signatures":[{"sig":"MEQCIFHpBiy6fLmNYsmD9Q8LGPyD5tTBPOssih955DTaq0RGAiBr2zmsgTvA8Dma6vEv7ABoZeL+MVTLNEH/fB4I+yZlMA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210920.e4d38fc.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210920.e4d38fc/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210920.e4d38fc","@microsoft/bf-orchestrator":"4.15.0-dev.20210920.e4d38fc"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210920.e4d38fc_1632122659281_0.8574982512353255","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210921.fcb2cdb":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210921.fcb2cdb","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210921.fcb2cdb","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"0dd1c7ed69474f087908fb828771c63472fe7288","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210921.fcb2cdb.tgz","fileCount":36,"integrity":"sha512-DTEeNWEhe1AfFg+4arDuP176MKwenRu7p4g0fC5/Oj+mha+WIRkzbPttDJZDfY9zAoPy4+P+P+2GIhu57A3xYQ==","signatures":[{"sig":"MEUCIAcJH7EW1Gsm/W1dGGQY/3CEZaXqEdfegt/Np7USiJrwAiEAvdRpxbuv308xCar5wXLKIShIpwBD/OAK8sOdQmCKG7g=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210921.fcb2cdb.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210921.fcb2cdb/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210921.fcb2cdb","@microsoft/bf-orchestrator":"4.15.0-dev.20210921.fcb2cdb"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210921.fcb2cdb_1632208991513_0.1080088405582944","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210922.c1c0528":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210922.c1c0528","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210922.c1c0528","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"87c33092ce00fd3bbbfbfd0ccb68c759d886ca7d","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210922.c1c0528.tgz","fileCount":36,"integrity":"sha512-PDyiJ2ckZ9Qe4s215yjCVJrZDBwD7+6kPdQfztqZOIcK0iY7kwwaSVov4kVmN1DNUX7PLrfMjOz8rr2ANgN4bQ==","signatures":[{"sig":"MEQCICKwCxBta+PabR5oDT2Xj3yEGFfkno3jXwCELW9tWpfpAiA0UD4LfkeCzkRTu9uz+xTswjjMfHSM/6noD0kNWFBXig==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210922.c1c0528.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210922.c1c0528/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210922.c1c0528","@microsoft/bf-orchestrator":"4.15.0-dev.20210922.c1c0528"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210922.c1c0528_1632295344515_0.9956359108752868","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210924.20a3f98":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210924.20a3f98","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210924.20a3f98","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"0057cf9f73a82d4c81afad7b1ba740ba29b369ff","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210924.20a3f98.tgz","fileCount":36,"integrity":"sha512-hbnTJxQhwBXSXdq/k7tSThe9CgXohH0G5pMJAh0Oo3XoWH9QbZTSEVqRr3N9/NWYBHgOY+fHX7H+HRdlPgKPNQ==","signatures":[{"sig":"MEUCIBujwKXvuSbHMoUfCedrxq3LsFsSXNymjEfFl51Y143JAiEAwJHIhlVXyG7pkcTUuDmqY9SiOX+hoGQIQ/08JIm1gbE=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210924.20a3f98.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210924.20a3f98/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210924.20a3f98","@microsoft/bf-orchestrator":"4.15.0-dev.20210924.20a3f98"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210924.20a3f98_1632467970387_0.2565045458251627","host":"s3://npm-registry-packages"}},"4.15.0-dev.20210928.a9a92a5":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20210928.a9a92a5","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20210928.a9a92a5","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"ee925d3c1220f8b4ae14a066254311ca60f21478","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20210928.a9a92a5.tgz","fileCount":36,"integrity":"sha512-7CKK+Nlrfaf7NKBZHQgWWVyztpJ0KpvfDTTjnjxgxTbtnbvaBahoM1HdEwfkW+GUxvR1YOHUe1fYTrMMG/Y9MA==","signatures":[{"sig":"MEYCIQDQ7opj3Hh4BQo499GZyZySZ4wL+RtvOeVzYlMK2gys7QIhAMoEfNwDlnujJR/PUBmKtjeVtISMNoIH06CwMtOAbl2x","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20210928.a9a92a5.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20210928.a9a92a5/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20210928.a9a92a5","@microsoft/bf-orchestrator":"4.15.0-dev.20210928.a9a92a5"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20210928.a9a92a5_1632813777078_0.4784870553130829","host":"s3://npm-registry-packages"}},"4.15.0-dev.20211004.edcf63e":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-dev.20211004.edcf63e","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-dev.20211004.edcf63e","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"179f57ea0c3f29234643f3d7138e03a9fe20c072","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-dev.20211004.edcf63e.tgz","fileCount":36,"integrity":"sha512-LcB4vuA+oK8TXTunI1fKdcTMbjdMYdYe0Gj8lLpISW61x/NAk324sIbeJrpWr1+wp/IScbbAnTDIQ3jH2Rhm5Q==","signatures":[{"sig":"MEUCIQCpQvQiXsQWGlp0DFHU8lDuL8KE3qIgxbcg7CfRzQhKhQIgYn4yYngZIL1t3tfBu+VWCww4a7lix3vvoKCAKOun870=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.15.0-dev.20211004.edcf63e.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-dev.20211004.edcf63e/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.17.6","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-dev.20211004.edcf63e","@microsoft/bf-orchestrator":"4.15.0-dev.20211004.edcf63e"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-dev.20211004.edcf63e_1633332323457_0.2928023712967587","host":"s3://npm-registry-packages"}},"4.15.0-rc0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-rc0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-rc0","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"06cd59fcccb1bd7f987069d6a30066e7161ab0b8","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-rc0.tgz","fileCount":36,"integrity":"sha512-PYXylGGAFtf+4ogBoskQui9bIYfENibMsbP1RTOaiVO8e6GJOGVzYpHPTQj7+mOwxI5i5FKH8CXBRZSPljZLBg==","signatures":[{"sig":"MEUCIQDwWzDcVrcGh49xhRG0ntjgLVt5zTHuC5et+kU1kDlOdAIgL2/8cO7qXRvWGYkfqSzK8ogGIbXsmXdjlS0rQ7Z2A4E=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130666},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.15.0-rc0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc0/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-rc0","@microsoft/bf-orchestrator":"4.15.0-rc0"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-rc0_1634338350897_0.9790232770733509","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211020.406dfef":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211020.406dfef","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211020.406dfef","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"7a5445b0433a3beea1f9ccb937e79bec186ac070","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211020.406dfef.tgz","fileCount":36,"integrity":"sha512-eGTiCe1QhG2hgahHstqZU/peQqm8G+4Hnbx/oYgO5OMNBwtpPKRVcaslLNj8cpxaZNPgl1Hzv/tftgDMwy8N1Q==","signatures":[{"sig":"MEUCIQCdSWE3yoATw31PHE5BC1JnlB/THd5ma8u0u3knbY4x9wIgSZXFGu+KLwfSMw8jpwgifx1KDK76TYgVzL3RJQwCrTU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211020.406dfef.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211020.406dfef/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211020.406dfef","@microsoft/bf-orchestrator":"4.16.0-dev.20211020.406dfef"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211020.406dfef_1634714638232_0.9204028126663708","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211101.bed3c05":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211101.bed3c05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211101.bed3c05","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1d50fc4d4b12e979c9fe9fbaaa075d871bab0cc2","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211101.bed3c05.tgz","fileCount":36,"integrity":"sha512-sq2gmZCs1j7t8IvTAE7h8TvYkvwxm1DM55djqMJId4GfJnZY7ocjqGC/otaXmJ346VlJRctcCgKD0f1Skx4pAQ==","signatures":[{"sig":"MEUCIANcHHnGFanzndcXr7/ZvhHZzjQ9SuRHewWPbAswUoYyAiEA0+0MAwE+g7bfPRMnlMiehmBaWaE+mrgeD0kZ4k+P2w8=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211101.bed3c05.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211101.bed3c05/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211101.bed3c05","@microsoft/bf-orchestrator":"4.16.0-dev.20211101.bed3c05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211101.bed3c05_1635788145482_0.24814500837630726","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211102.bed3c05":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211102.bed3c05","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211102.bed3c05","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"523b78cc8a353a611cf13c075cd61ee7b56014e5","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211102.bed3c05.tgz","fileCount":36,"integrity":"sha512-jetS4IQsrHfDMhxh1XA/08E/pOnUgnaCM9fceAPkoLNYssNo938Qj//cD85PrILHBdGKYNxnclxfcqZQbcwugg==","signatures":[{"sig":"MEYCIQDDHVGgdCuQmJDEbbbQcx7bYbM3NbU20oSaboUkEJ2eaQIhANfSq6wPjFSSyctyNPl8clmNWwk1UcI9gvuWlVxq73wt","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211102.bed3c05.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.bed3c05/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211102.bed3c05","@microsoft/bf-orchestrator":"4.16.0-dev.20211102.bed3c05"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211102.bed3c05_1635837893459_0.15428022925177465","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211102.3f7a67a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211102.3f7a67a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211102.3f7a67a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"212705477dbfe076a95900fcc78ff1c53e97f63f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211102.3f7a67a.tgz","fileCount":36,"integrity":"sha512-ITzTcxKtSTjSScERvn1bLTa61xe5t6s2halPzv/EGMKA3BhVTgMhl5Pmut9AxT58AIl+B/kY+sUbvPkaceDOJw==","signatures":[{"sig":"MEYCIQDvRTOreLIYK4q2ZnWm5X6aFjeSvvmHyd1Djk25Zo9fMgIhAIcax5VOA2lTohGLdAJn2cgTcjFTPiqCh1noQzz692kX","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211102.3f7a67a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211102.3f7a67a/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211102.3f7a67a","@microsoft/bf-orchestrator":"4.16.0-dev.20211102.3f7a67a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211102.3f7a67a_1635875307930_0.0641919553718584","host":"s3://npm-registry-packages"}},"4.15.0-rc1":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-rc1","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-rc1","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"9c1bb347279628543b145ceb60a8405e53134a06","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-rc1.tgz","fileCount":36,"integrity":"sha512-fpN90C1PnIgMiYSVZHjdk6LrVwtT1kcs+0zA0egPNTRKPgnR0iBm46MimmuUnk7XVatxB7gk1xYd3YbdEA1gpQ==","signatures":[{"sig":"MEQCIEdq8Iscg4vr09/kM58QP5j6P6yI5I0BkCiwXjH8MViVAiBejxBJCceTL0R3cb5BZlqY8B2cUN5wBYiZkr0vpcZfeA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130666},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.15.0-rc1.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc1/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-rc1","@microsoft/bf-orchestrator":"4.15.0-rc1"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-rc1_1635895057770_0.9314192639922585","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211103.3f7a67a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211103.3f7a67a","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211103.3f7a67a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"cc775c1f65bf3a65bc825f4b452d751398aa2def","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211103.3f7a67a.tgz","fileCount":36,"integrity":"sha512-5mERzhYmcYJDEcgK7KaoZJZg3b6P0cOR/yWlJWHwE8lpFe3Qvs6FU/9XQHEsTPjlV57be0NbX0sQf0fPX7lYpg==","signatures":[{"sig":"MEUCIG0YNKC0IWtsGm7TbWoNtOL3o+3/GCFYSxP9U2N8kc87AiEA1Y0CsivZy5/CdxnNwKJ/2FgIgle8/ZvsxodWje7oV+Y=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130904},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211103.3f7a67a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211103.3f7a67a/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211103.3f7a67a","@microsoft/bf-orchestrator":"4.16.0-dev.20211103.3f7a67a"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211103.3f7a67a_1635924029204_0.6279810932649712","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211105.90655c4":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211105.90655c4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211105.90655c4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"e2da26297c85ce0f8318c78ac4a753e35e5ad204","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211105.90655c4.tgz","fileCount":36,"integrity":"sha512-qTCVRDjnp8d0TuV2UJ8EHpYvNHqtRFCXnknxIXOib/Va8GoMVXXZt5CWWkpRsjPcrB5aJ51M80KQPJ+Ex/S18g==","signatures":[{"sig":"MEUCIQD74oT/e/hzewleFI0Io7Fm1dDgGSynT34E37L8vi3JogIgC2awgckp81GPKfhErJpeMcy1VUtiYA2qkWz0fIEObmk=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131571},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211105.90655c4.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211105.90655c4/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211105.90655c4","@microsoft/bf-orchestrator":"4.16.0-dev.20211105.90655c4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211105.90655c4_1636097104565_0.5030020230971217","host":"s3://npm-registry-packages"}},"4.15.0-rc2":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-rc2","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-rc2","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"527913c8993eedc894c8dcf968b932849a8a4151","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-rc2.tgz","fileCount":36,"integrity":"sha512-YZJvJKy/8LD62ctm8OsgDioiboF5beOOo6cNpYY3ItMIMU7NTU7FXwCHJzEsFVl4P/+sm+aDuUtiRGg4fE2C4Q==","signatures":[{"sig":"MEYCIQDnNMkKh89TmFCnh1rv28IGm54flNv2UhjNDUoZxmC+MgIhAP9ynGgs1ivahh9PMzqZOOMulFq27EpooL5OtslF675v","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131333},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.15.0-rc2.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc2/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-rc2","@microsoft/bf-orchestrator":"4.15.0-rc2"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-rc2_1636136498086_0.15013525117098525","host":"s3://npm-registry-packages"}},"4.15.0-rc3":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0-rc3","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0-rc3","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"0ea9e081ed731ab1b604cc99d715bf0947720c74","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0-rc3.tgz","fileCount":36,"integrity":"sha512-kwOSoVYSkBDQulbm5IwFwhNSw94Jr/XYgMYtP/zOchP5u2HlLouj0GcfnKHcMf00AR5u8HInHAdcQAc+/EP+zA==","signatures":[{"sig":"MEQCIG5EoKx7LyzyJdJi7cbfON5GVmnqq6F4hYLpDr1g8JgJAiAPfo7Ll945rigN4sNaovWNh3MygQqJK/cyQKJg7aCjCQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130780},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.15.0-rc3.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.15.0-rc3/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0-rc3","@microsoft/bf-orchestrator":"4.15.0-rc3"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0-rc3_1636587635234_0.21497496409190187","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211111.15b5c37":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211111.15b5c37","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211111.15b5c37","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"0150f834be501201822d458c7cb67408851700cd","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211111.15b5c37.tgz","fileCount":36,"integrity":"sha512-tcOavRF2j53KEEkFWFntLrfXM8EM8PifljHwULTv2DdpppBpK6LqEj7N3e941w+2FNqp8ICQABcKcHxFfiQaEQ==","signatures":[{"sig":"MEUCIGnOLmxY1qVHhJbXR1yNloB51VJVMoeeptrdxLPAMa5YAiEAgbyvoygrLnjjcp7DkQqvF9Q6JiZ71oVdn0GQ3fz1QVQ=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211111.15b5c37.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211111.15b5c37/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211111.15b5c37","@microsoft/bf-orchestrator":"4.16.0-dev.20211111.15b5c37"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211111.15b5c37_1636618957121_0.5534510488876083","host":"s3://npm-registry-packages"}},"4.15.0":{"name":"@microsoft/bf-orchestrator-cli","version":"4.15.0","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.15.0","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5dc781a3cbf049e203d7103d2980cc319e3ea36d","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.15.0.tgz","fileCount":36,"integrity":"sha512-bEYU2Nt1qUfINP1sDV4ukSCbdTESj00wWgHFnxZ/spnoH/Eg52O3LB2YkBKNN0rF15al4m4rkOy8PnTLeK22oA==","signatures":[{"sig":"MEQCICo37EVsb7Pc+WYRX/LlhpqDqVybNbk0gtbHGpEEnJPoAiAa0Xvazq0mNyvuqmjwh1XI+qwgX3MV7BTZEkFJTPK56A==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":130724,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhk+ZYCRA9TVsSAnZWagAA6TMP+gIBsJFitnf9eFV80Ny7\nJDoaQKpKtreLFCE/KUKHKATxuhvHUHTuBtg7kBILjh/ek1YcG4E5AkT8ArMw\nzX13176mAHcLRCvFSL29k2Aygomnhm3JYjusbJwcG7YSuPpfqFjIJ8YzbO80\n6YpUvhPbirVUp/Rym6CFu+DTZRQBaBt+dHUF0Hsg94oTQtmA0wowlQN8EXQj\n8tvgj7fiB5ke8pQTrIGGuF1YlnE10/sQZqBmDLBQ+/fgT9EVdXoU5DnGIoGl\nQC2RASpkmDA9UtuS0AsT0pcG0nMS0H+6O38nfJD9JN6mgORsfA140Fj7S24V\nhYIyGeWe33OZvINafsHdQKluLmoXAoN+YDbgi10v5PPRXb0vgiuXYziXmjBA\n1HT2hxYAEUnDP+uaF1VS1s4YKV/CGgrZLDFs9gm1vwo56Ge18dg3n0iBWzdZ\n6Cgx7cQGiukWi+/SvPiJFeOcT/pZkP+yCW6SMzG033kqgDcCCDt59AzyEwXH\nKxe1+OAF71bzkxRP3CWFSfUQxKt2TCsw3ZoSdxT7VRv26j2JDUQNADnt0aj8\nmIc7HYFEyfZi04snaOc9DpTYFTg9UnSZZunTD7yy7076EMHkYUqS+BOxto6V\n8BTjCJVw58HteOAhF2bykFdEZGxzVHQ5lDTsqLxszrWw3ZXS2TF+iutyyiKT\nz1UM\r\n=9klW\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-rc/drop/microsoft-bf-orchestrator-cli-4.15.0.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.15.0","@microsoft/bf-orchestrator":"4.15.0"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.15.0_1637082712283_0.9606358929247036","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211117.19ff452":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211117.19ff452","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211117.19ff452","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f095844b26a5a9e08a4dc4773911598635d258ac","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211117.19ff452.tgz","fileCount":36,"integrity":"sha512-s/KCJ538I9v+Gl2pGuee1py4/aXEb9rQWbBM/iraj/C3IhnQkc7yR5r7/t7jgrbzUHEFe63y6DpV4I2hj6N7/A==","signatures":[{"sig":"MEUCIHgX84iYLz1T/EqBUHhIptN2ZSp9EatQJOK1cdKA2rOiAiEAzi62fPHXj9xgKF+GsyAjVC/DlsGJkCM8HS0H6Ol55oY=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhlLwbCRA9TVsSAnZWagAAp7EQAIV3QorLBDN3lw45ujet\nMWz02FXcSSLoRfaoUwpvrXwx4c/Z8E1MQbialx7jU5DZqzF4FCRZdWwawXf/\nYZYZideS3p8ZaclV0CXD1sP1fxwGNqhkkT5sK7k2ma1YKORtdrjBMqsCoV+o\ng1cYvTK3kPxgeCir61Wij1nRqcM7c297vApVJ6LvzYBorTi/QtIF/Cm1To5q\nJ0Qh+xfImxbYXdekmbp0EsTJmGxfwemlKFHwn6nm0hfyCcXZ03VClEzEVUr7\n+IAuoS05+vV9AdkscWToquDD8Z3E3rV2ida+ok85mKdAqc9+gcSnqBY9ySRz\nd1Koz5dQJFIbcYGUYIO3tBfd9hT/r3bYZAQdhoLsJ49IUer9z9D5XrQ44TTT\nN2ABkXt8F86gYKJ4EaqfOnhWmJNvUdSHwQmh5KCkFBivfBoWKJ6hThRSzu0q\nUgU55zrHZpAlWLk3V9+Nv+UyE3l+I6PndqE/XkES8PfxQ/kJCyk1I0KJ/u7y\nXhmEIRAAW6uBtf5SWrgiBKS2zZFJrN6dc/nMuBLHaC5dYjUT3/wgzP38Il/X\nXAZAJGZ8d9TnFsv+uP275QyVLAVf25og84XWtt8IDWryvdT6d3rAy5aTob1g\ntb9N/Pr33XdzgtX8krZM0vj7XnqfhUwoRavyDPgv80axbbb9hJsxL+oAf98o\nPGgw\r\n=/pT5\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:/a/r1/a/_botframework-cli-daily/drop/microsoft-bf-orchestrator-cli-4.16.0-dev.20211117.19ff452.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20211117.19ff452/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"","_integrity":"","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"6.14.15","description":"Bot Orchestrator","directories":{},"_nodeVersion":"14.18.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211117.19ff452","@microsoft/bf-orchestrator":"4.16.0-dev.20211117.19ff452"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211117.19ff452_1637137435251_0.8443284855346223","host":"s3://npm-registry-packages"}},"4.16.0-dev.20211220.6972973":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20211220.6972973","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20211220.6972973","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"97ba09a726243a98f1627db43145c10d596eb92e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20211220.6972973.tgz","fileCount":36,"integrity":"sha512-miNnMhYPFACz9ALf6IsWSBq85PcoEXCyfMJx+ZMVj4r3EYX9as9oEAH3AAumcarJlXqhh3Vm5kFLNNEW3P3/mg==","signatures":[{"sig":"MEQCIBA0/2mD8/5a+E8t5lDVZDLIgrzeYdzq4YoNFMdWSgAQAiBW95HzH88M5r70gXfO3l5J1DUU0k41VQ7IHUUcwN2OJQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJhwD2zCRA9TVsSAnZWagAAZ/AP/3ZbULh87wooHGaVA707\nCWAgEkeaPBBVFANdi/q9IkDQ0fsSNi/m+kCC8FxV0iC1pDczIA2hLX4RXaAe\n/t97Olw6/+t5cLWY88O9jBr2o+BOOCBGQqlu4AQ69BBK1ZVC7zhg4mbg1lvT\nXMYi5DGIx3mUEVNyUlBnQpAU1Mu5aWQ4pKjK4ZqVKsuVaQQmIWsreyIC7WrY\nc2y4VM9tLc4ja9Jy0L6oBu7LH0cs906NUhY6gLhf8dea3zSESl07YlIZjH98\nmsBuQbkhmIddj40r6VOk76G+JaoF2txXlUYo3paS+dzSe6SgrEMA8bd/6AfY\ncgSUVPc/BinlSxSZoGFaA248WejMMs4cym3m0Bg5j8xfA2gZoFYlOnGhHxcm\nbf+eCUQwoezPiYhaqeICZ1VgvD9n+xV4g09ToKRUDewxIRLmASq9ctWaDhLD\nalVPobd0sehL6Z3oVvr7wN8ZzMbPyj/gt0ug2Xv5+MHNJjZ2grAkJZ/VHjkD\nzkm0qRIRAOi2tweX4J7KgZPY8/I6FXAEBfx8lv67yuP8wgyNfw6TcJdQGMu3\nC8YvLZ6Gdq6IfE04sVsb6ufsZjdlJ2vUSoJgLzxlQmccwXIAjtrKlu+G5xmx\nqD0EbWwVxrib4s31i1MwT0Uz4tUaUyovj+6jBOdQXXRKVwbNbpGAMhBGDGE2\nLT1d\r\n=YUZN\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20211220.6972973.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20211220.6972973.tgz","_integrity":"sha512-miNnMhYPFACz9ALf6IsWSBq85PcoEXCyfMJx+ZMVj4r3EYX9as9oEAH3AAumcarJlXqhh3Vm5kFLNNEW3P3/mg==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20211220.6972973","@microsoft/bf-orchestrator":"4.16.0-dev.20211220.6972973"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20211220.6972973_1639988659071_0.24278450601943935","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220104.33b5e0a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220104.33b5e0a","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220104.33b5e0a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"48e4bea557a1213e2669c86e7e5fb2868ec45ada","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220104.33b5e0a.tgz","fileCount":36,"integrity":"sha512-+5uX9/6POqWVtOWmBex1XIYpY0pdBgAynOm6wLFutBt96r9alcLiqN0Tohflk4DP7lPiNxQ6QSeFO9QpwZpjbg==","signatures":[{"sig":"MEUCIDbg+GFaf0PN0UvB1OGSV4tNffCReaE6FZ0oONS7QBE+AiEAvxxokNLxyfNAfyDAAC8Y+kY7o/r3t/OWqeEPhkWRNko=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1AQiCRA9TVsSAnZWagAA130P+gLhdwGBHSJQGyVEZDe5\nlGirqkBsWqpUohqxNHLyvuHhrNLc+IYskE0St9amLPCXaHeOc++Zvp0ONY5a\n+mYA/i01fl55hNK7L186EO7V750K0nYfTXLXkvqyQdSD7GpxoTj9Da4pKYYO\nkAOY8B0sjR1cMxA/BfcRn4xQ6x+FL9+5St39t73qIgDIRi1mq7vZ7BJUltpf\neKnmD0Q0bNWvJeEvqUkLrv+vLKl//k649QpLJs54CAoscKafBvyvtZo75rTn\nENOlKiIO6pl5m34P7/jGsy3Lfn76TXnZq1z1naE4GUCoVTTwKJQg93UUdG6b\niSHHY/Uh4QHtwgvtgDYGaqDaDyn0B4E5FtV7EufeQCbJPRe8i0htyd3FPcKY\nrHL9kTlxFxCij2B7rzvPgi74ydW0AGWGp0ygK92CjO0dBYBxNzYhTw9h4FQu\nqyeCbHZxzcFQoYPj7uOoWJKXeH9F8rclMCRqEL0E03rkDinMTwiBpBlL9lHE\nWtWqQHH1BAG9DjN1R1U+6Woju0dF+V6I/Q35nNyQ34XWFOtEn7B0zoCmyPiZ\n6vbjmkRPQfZ7OhyS8YXjl0jy9mZYAsgd6rjl0f7lRZPJL06bfi5wUH1lcRP7\n4SuoCZfrMLwUXOHckHFyNEMgEX62PnKDTf+462PffcMdaVplogruUI5CMbPZ\nN+Re\r\n=udre\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220104.33b5e0a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220104.33b5e0a.tgz","_integrity":"sha512-+5uX9/6POqWVtOWmBex1XIYpY0pdBgAynOm6wLFutBt96r9alcLiqN0Tohflk4DP7lPiNxQ6QSeFO9QpwZpjbg==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220104.33b5e0a","@microsoft/bf-orchestrator":"4.16.0-dev.20220104.33b5e0a"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220104.33b5e0a_1641284642178_0.29805729838140893","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220105.380511c":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220105.380511c","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220105.380511c","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"cc070b1bab8fcfb4f50a12bf88f966cf6ef8e308","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220105.380511c.tgz","fileCount":36,"integrity":"sha512-JOQrAvXEf9euKOVXYoyjFa9lAoqVuegrv2So7dB3CUe1PE/FmrZIqyyjmyMLodYSEEyjanHLEAF5Q4d7QybjIA==","signatures":[{"sig":"MEQCICJQnCGrckJm/FDkn2GkHm5T3bWa+6POOOGHrDVEjqvcAiB40kkBfUWpqxLF6T6IatsTCVkNZuhdTwlta0yfCLYmuw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1glLCRA9TVsSAnZWagAAVpUP/0LyHzu2nCBS1XunziHh\ninqC004L1cbes0zccJVNp/Uf2r5VGBs3JEkw8X3XC3zc/N6pUJXsOiQHHGO5\noOYoYO4HGFXXHY+YP5pWcgrajH8FDDZdc2ZVvavfbMSGKvkS97KfdwVsukbK\np3V2sX9gHr6Dyp2ve/Ooo6m0WY4Y/Cw3eaMVHey0Tjqus5o9LGxLkI3OpIsM\ntXseGEG4IXP2wGs+q/P2Ylln3TdynpWxPheamHJIC0GidHF6nM0kWPAx2+Yy\nMWr+TMFGqwc3pM0PLa06Y3Wo7MPK3jOlFtb6AY/4QE9e1evsd1uUuSXcbjwn\nQ2wGBl33VkUTdAN35atrKikFyq18EtNJMguTHUBKmwjfJJ3K6U4as//howjt\nhzlEI4mYi+ybvy9ptJKQ3vSZsXd8L/K6NmYFb7hSYnJ5O0qHH3OLbDgP+TjB\ngGIx/cX0rn0AMxV3nDia2JBPQT8MwHruHFdULQ/6bs1vR7OFQ2FR3eyhUYt6\nokhbe5LN/95Sh6mBwzN3VNbW1yleDBhoj6s3nFT6w7ZmIIVcSf8XJ6vDr/ZP\nCSRK0lxWMgY4uWXqly7iRW3zHJto3olVAE4s26Fr1nN3tQUUwH+p0L+C1rQ/\n3nd9TDLWG5kMrVsKyA44YI8d/ywD2y0HsXs1/o/aXPkqNrkrvCC16d1qkVlW\nuL/X\r\n=UM+3\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220105.380511c.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220105.380511c.tgz","_integrity":"sha512-JOQrAvXEf9euKOVXYoyjFa9lAoqVuegrv2So7dB3CUe1PE/FmrZIqyyjmyMLodYSEEyjanHLEAF5Q4d7QybjIA==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220105.380511c","@microsoft/bf-orchestrator":"4.16.0-dev.20220105.380511c"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220105.380511c_1641417035230_0.4621522136752625","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220105.c44b989":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220105.c44b989","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220105.c44b989","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"50e589514f25b0ecbd069c5da5bd37b249ad144c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220105.c44b989.tgz","fileCount":36,"integrity":"sha512-uzcJNnq3Y6Qyh/t6Cbm8exZxUQJiounkkN3YCNqxBfNIo8U1iZ3auhq6LGWEpMzmYqa+tlUF7HnVEDuGvUu4cQ==","signatures":[{"sig":"MEUCIGFnM1uPYFOkT8T4jKrcuakyuHY9D8Pt3HijdIxUrRukAiEAu8ZdKXuU8N8PNwSZBlu0stDndWF6F3xyHVvS2y/oNRc=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1jGICRA9TVsSAnZWagAAMDMP/jf21ufto4l7ftpZokfh\nBSxvXW22NNKZujfomjAKUajs/FOydEYqM7mBZoYjOIXWv1PVim0EeuKUNuXC\n1mSbQ5rE6Q8d9vqR0Ww3FM566gsRH+jgHae0Z0uwrTooxX4vRjjr2fN+PVvV\n1DT+KuOzZ/zrN66XI+JlNDdzMMnHV9FRigA0WTG9ZXaX+kEHZ6qOhy41JInY\njbojMT9a7SSC7bT0Q8aD2tC8VKRqTEvYRsJgE+EkSPVy2HWdtr0FlzQ+Qin8\nf7RmOE3WmjxIw3UyIdXxlDyutY1ZzL+DlNPIYMsB2Z91WWUtk/6jxI3ayAg9\n54tWDvAO1XxqsOTGhoYOsyDgDM3WfdRGLzrvuICWMn58c6G+YCib1QHPJKGp\nphcO6Uydb9TEtrJgloQ5qUtpZhE9Htwtla3u7fMpqHw4LhjRIEwLiaQq00Ww\nGfAOL3mqVyeNgKFeEc5zAsVW3LIENov58aJCnIDPJRYhApqqnNwDNz/xT3HO\nUL3zsgvuJOZobh9Ab83O7DOuFr9zWRFEqLpAEmF9GuiPrx7rignVFmrnkNs1\nOOPhHc878OFBrV9vZOleDzAFdbWrQmV4w6jOF6P+EgWloOO8Mqr51jGeZOGA\n2sZqQCPYvH4P8B9bPMG9hqWHWuMna+xide+jhgCk0g/oEhPhjenvmOycXeZL\n/RPU\r\n=VMVU\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220105.c44b989.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220105.c44b989.tgz","_integrity":"sha512-uzcJNnq3Y6Qyh/t6Cbm8exZxUQJiounkkN3YCNqxBfNIo8U1iZ3auhq6LGWEpMzmYqa+tlUF7HnVEDuGvUu4cQ==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220105.c44b989","@microsoft/bf-orchestrator":"4.16.0-dev.20220105.c44b989"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220105.c44b989_1641427336775_0.2358402505962367","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220106.c44b989":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220106.c44b989","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220106.c44b989","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"6b312bd8587feefefff39a05f10c87b621ea9a8b","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220106.c44b989.tgz","fileCount":36,"integrity":"sha512-SxveyWlpPcOH5ljkrwlRQqucsWjOCkDR1JZl8LWJr3adS8ObLBcmZWb33LaGqRl0tMZM8YNXgzT0quaKnv0YUA==","signatures":[{"sig":"MEYCIQDYAGzlb4OSk/u8FWGqGE3vlgB6PgN2q21tXUunXzQHIQIhAOBo4ikXp2+Qrn8em4aTBpue6fIZnxLf0C7UH08Q8OP1","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh1qaKCRA9TVsSAnZWagAAdpQQAKO5FEf+GtdJavjgT6Gj\nL5DnyyKijwTxwxxnIz6VamE03W2mJ61ZLBAqLmwYNLhFL8DQXNi092nZndqr\nKarjPfEzEtn0mxksMWd88XQtxA5C0ULiOpaP1B/vjCtz0gXtM3qd3UbQgVv2\nLE8dGtNFuf3HrLYuzHzGgB+lCIgkbkwqEvRthEKtDS54qNIYdW7pppW//QIF\nWyZr/OtUI9BdQ52BCjR2fKT3MtLopGxHJoidxQ89P0QrXGXoNsnM43jsp+F6\nbyrDGD2wDqb/3f9niq5KTe+ylhGSe5loo7t5VFgBUEBWWGoGE9vYLgR0fGDL\nK8cut3SvI7riZi0t2SB7GbymgmEDD5U/1U4FAB+T0Gy3Lnf1/rceWnPynpBb\ncuUHZh7U5Q0aSnLt8Oq6xaRYHB50O89m09kiiIya+pZwX3nyQaM3xw4aKwB+\n+eOUWhTA1m4zlDuxOVJlm9ACV040za1vgvuQ2ciTBj066k0Kgp9x8n65iX9Y\nO5eJMRCHXAUKYXrsTWOo3aFs///FpsLMuAZZvypF2HOGWjy9uJ95ALPriGJ4\nUnE06jZ5IGjExYt7ujWTb38x7uV3jccVlbkAj1QVW5AcMLgwki3tpuh/ej5P\n40v0HutHQi8/6mF7L9S9/kicfv2YdeFuNn4n+K5SWTBkiGbLJl2xLWV098ex\nv1OM\r\n=/MPo\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220106.c44b989.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220106.c44b989.tgz","_integrity":"sha512-SxveyWlpPcOH5ljkrwlRQqucsWjOCkDR1JZl8LWJr3adS8ObLBcmZWb33LaGqRl0tMZM8YNXgzT0quaKnv0YUA==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220106.c44b989","@microsoft/bf-orchestrator":"4.16.0-dev.20220106.c44b989"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220106.c44b989_1641457289864_0.08032210548874086","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220110.b4d497d":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220110.b4d497d","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220110.b4d497d","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"a61117ed8b37612efb0bb15a9f009f2b84e573ae","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220110.b4d497d.tgz","fileCount":36,"integrity":"sha512-Ffle3xWa3x+IKoybrXeQg9AU4+1U6sicBIcttRAYiLM0Zmca/teGDowM+ehf2mng4Xws615OCHHmybZD7Qge7g==","signatures":[{"sig":"MEQCIEYnmaLbvPzEck/JdiZsR44pRHKVFO3NX8NAUxfKW5hUAiBwzDJuMUpQQXLTZb4Jxsrl6R/YHWpZpSenReCgut4rFA==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh2+zUCRA9TVsSAnZWagAAzm0P/jKD+VHsHjwlKatdKeob\n+1FGVHrSavVgJxRRQY5fZunZItxI7kfe1lq0VT21mmcZOPUhkUmpHAUHowEm\ndGf6I/WfiZJOjUG8cYgj27dAxV1uIS5rK4Jsirkz1bZH43kgz00Vc2fZJOH7\nRmUlDgApnUNgawSoLA3UF+kSt9DcTWqfFPna7EDQVzv+GN+iWpEHi8p1pwzf\nKLzM0aQrbCYceKJaUtlQLV+z9U4ii9dO411+h2uKn54hincKlb7dw95Hof5T\nytCER7mDOWopC7lXx7EzJ+5EIPgbSZKtBXAXYhnaWascq/gTPUNiBgoqeX5B\n8vhaBDl8Qf5nM93Nyt0qpmiVN8agjQUyuWd2wN+hx3FLq7bd23fY5P6Hk5Me\nr73x2EUh+HVhP1Uwoj5tzjTlYDMAZWogJTtHmp7fSsYFcwMwcj2+OkDZoCIN\nlSZ+xHRd0n7EVueMEcRgCs0w3MAsprQdQCf5rpgDp0fAJ6Nh5m9A8crZR/Ne\nDddYjznOFeZLwlHiLJlrH+QpWnRbTNP3Q/LI+YqnMQC6ZKDgnZYJ2Avdmzhd\n4ofNIU5Pjwc8FiqM2jEs+1jLjPQGxcWc0MQN98WPZwcUFgUoP3/pFSbA2Ac1\nZknsK+2Tmf2Il4PoliXPIXVaQ0+2sqZs5+Hbn09yZzYISnRH3FTu0pcmMr8i\n8t1R\r\n=aNf6\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220110.b4d497d.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220110.b4d497d.tgz","_integrity":"sha512-Ffle3xWa3x+IKoybrXeQg9AU4+1U6sicBIcttRAYiLM0Zmca/teGDowM+ehf2mng4Xws615OCHHmybZD7Qge7g==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.1","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220110.b4d497d","@microsoft/bf-orchestrator":"4.16.0-dev.20220110.b4d497d"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220110.b4d497d_1641802964331_0.3759325013020085","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220120.e933dfc":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220120.e933dfc","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220120.e933dfc","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"152eb51f8d363d2cac8b3c56683a1a61fff07b11","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220120.e933dfc.tgz","fileCount":36,"integrity":"sha512-mquRJ/htK4BX6XEXKKTJxCFqG1FhWLnqcy+kwGjKEcnv81DyX+ReovoFNtlgbXYz7bE8QvxTfNM6GhCFMwRV6Q==","signatures":[{"sig":"MEUCIDCmKGxdHASv2M/6IzuqNpGCZllecwUwi+GHD7aYnacIAiEAzdZQXVLzJZBn9XjXxtsYDJdB2C6/tNGZOYfPRMar5mA=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.13\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJh6RxcCRA9TVsSAnZWagAAhHEP/1Ib5VK0vgeHRa2ixzPZ\nYChExIy7NPzJdXsVy59iqu5ZM4vmgzqWzvGdIpvq1fFPdu6dU0SdbXyL2vq4\nRmlQqsXUOeZVtTqs7CnTXQNI7U+Ea+JnTOPOv5bdMSMC1yUr/V3a8SeMCKLM\nkbK6QejxaVfPYsntbKwybC5+a0QhQdO66uUUCzmMBYI/7ouWgGnSdBGkhgra\nkt5Rib18uTN3d5PGGl90I+NOPUrkx7s/WtEoU9TMq1QD4U175Oifz5xxUBSC\nStKyknAKM2wNSaiVbgxdD/K2L2Q90XCIbWwsceVjyxsxgsyRvvab1D2Z/XSe\n4Q/YeQMMXlJkF0NEYkigAAmOAa58HmRpCBM57befpgtjMJsKKaG8Z2aJqQJ9\nsS3Mm9SjtORndQBIrWU2dUWo6ofS6rQ8c/BzA8TFC/XH6ADyOpVg1IP/tC7i\nJu6e5Hncg03DkG7SkDIC03iFOdS5ZJ0/U3wQ6HtQTAfu3Xtw+cA4B1MAhy6P\nupRUI09gbgwUTA+zJsR66TfhEjVAc2cmYtiA1zSBt8+pIauGxoqrEAg8GNSl\ntb7D4qB4HKyJAkZjRmMLbCIHynwt1X1HZn375GGX4+lCbRbldndATpBQ6FaU\neLhvcvvV5hUWqzCVhoXSyCxhVIVQd62DOHrtP2uR94OepgK7PsZMsi/DpetB\noKyu\r\n=SeY6\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220120.e933dfc.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220120.e933dfc.tgz","_integrity":"sha512-mquRJ/htK4BX6XEXKKTJxCFqG1FhWLnqcy+kwGjKEcnv81DyX+ReovoFNtlgbXYz7bE8QvxTfNM6GhCFMwRV6Q==","repository":"microsoft/botframework-cli","_npmVersion":"8.1.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.13.2","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220120.e933dfc","@microsoft/bf-orchestrator":"4.16.0-dev.20220120.e933dfc"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220120.e933dfc_1642667100759_0.85578514670431","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220228.5bb413a":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220228.5bb413a","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220228.5bb413a","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"f516e2dfa0ad921e5d8c44aa564d732d7247a7c3","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220228.5bb413a.tgz","fileCount":36,"integrity":"sha512-UBdNXcOq+D6GLf5dZebHCtfdk9tHNCHjuSpZlC65N9+yO/rwvEpvuD1h2jamO0N5rFKLmrDSSDIwnjHyW1K6uQ==","signatures":[{"sig":"MEUCIHigjGXkuC48nPnUIw0s/bq3ef5PgSd51jH3CFK6kZYdAiEAr047S609+V7x+ARKRRDMcBsWCXDtk1vcr1s80Q7oyNI=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJiHIX5ACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmrkcw//ZkZlT5mXdLAolruR1yEH7FTDOziyVxTNQz9ZskDAoS4qeJst\r\n+V4UFU1MUxb3RgHupOdrVxlEMLhSypMo9/rP4lAEQV3nbvVEiBHJwqCTf1cP\r\naX3LDM5NswGEB4GcmNISqS3CHonLH4dNZjChjkpsZTrXr8WMfJTfLiGHpLuL\r\nEug4i37w3zBJrEV/TfEMjtj3lM8FIhrdwTNanIw92w+Ue+tiVX4tEpf57kh3\r\nhO8CUdHutCLf3skoVk6ZKfe8Rog1sSIZ5FXhtsB++kZ5w6SW+Tvmm/JsIfrM\r\npSWfSI8Tj0WyeZ3dLqPnrzm2+2otlvPJmJYU+mIi3jOh+mB5nsPuHh4KeYGt\r\nLdMekcmseV0IdCUS1+6HU7ggFsHOZudDKTIU1e50789Cju+RHMSJ/qeep+4f\r\n8DVunPMCe2DEjK4eNXTHadx0n+5Q1qJqhDDI7KUkQ8CF9nUscC/KJ0Z5kgXU\r\nZ/JGVn/WUUmhcpKSa6eg+Bi6i8MjUH5e29sk+Ci/XQ3qh3/saa9jYQQPh3Np\r\n7v4hhzd7/TDzv+t+guGruC8sYgUP42mC0tBk2Ac52cguhRLC4vsV7nsk4QNj\r\nz2jWiUYOSZb70OR06C7pcF6CRPCOl0GJMOAcnzMZvcFCiU5mvffl37cnu832\r\nmZiSJ4yuTqIn53HnuMcBGxJ+ndTJLHMWaAk=\r\n=oivz\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220228.5bb413a.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220228.5bb413a.tgz","_integrity":"sha512-UBdNXcOq+D6GLf5dZebHCtfdk9tHNCHjuSpZlC65N9+yO/rwvEpvuD1h2jamO0N5rFKLmrDSSDIwnjHyW1K6uQ==","repository":"microsoft/botframework-cli","_npmVersion":"8.3.1","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220228.5bb413a","@microsoft/bf-orchestrator":"4.16.0-dev.20220228.5bb413a"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220228.5bb413a_1646036473615_0.3853139164870305","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220525.147c370":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220525.147c370","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220525.147c370","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"c9e4ce63a67434b8daabd261d37449cf25f695ff","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220525.147c370.tgz","fileCount":36,"integrity":"sha512-gRRPayVfSvWw4bBQMGuaI3lys+pXsc7DhsDdDA6Q/XxcKtZ/+NG7kVvNneRQuDcl3/DF8pXCnyAkuBC+e5n3zA==","signatures":[{"sig":"MEUCIBPgaPdHeV4VdK7j8DWVQTtaHpd1OOVqxKcKcIa3jaaGAiEA//MU5F6LVCMjrqf0hDSQsDusZL9tzN1zoVJFk4GWGcU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJijdlWACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmrNWxAAn2BjgwZL8UIDE/3G1/tFW7caBkxfeEJvpydrmiYm2jGnHL+H\r\nsTQWcTQlakzpLfPk2d+8txK7qwhOlcDzubOMLwlBFw9Wq7l9pCPDoJPC1jvZ\r\nldiuO/5CDHww50tfis4SGkpO/pKLL83ITuh27LEI6uE4Af6Hfc3cw3JKMP6Q\r\nAJ/Jbf7wKMOZXaKiJGdnERpS03fpXowq8qOS35GKHj8f6ukIEcDLsRLTlxzA\r\nc2mgQ4K5tyEZqhXhCCXC6uFaGi6o52s98MvxwdhC8DdNy1Mdiv0bYmteB8iG\r\nOTyyKO+CQGTg706czTgOeTrhPnlghsd6hj6xo3T3FYETMUTjQIGBZ+UDuN72\r\nIeUw2ip1J4y/FkvdHYxmT5B3IipSrChn4Np9shmgpXbfiP+9t4Vc8POvOoPW\r\nnJ7eN0PyQZyHH/Sb0j+YjNcb4S9DnvKIZTonmSvMMGsRh0OprsJsBNb4kn2L\r\n6I0HuoWO1Y9iIyh5Ynesgg4EcWH6z7sSQfIQP7OoRDXyMzBa1I0+qElxT10/\r\nocfVpiggiPgU3Gu4UfCelwFbKD16D5dgBIEpaCB8/zenuXhYiNZPhFhHXVz6\r\ne+oy7rGyAx94DNhBRJKMrDHuv/PxjeZPqPFH6b24Nej+e5kTA87Czf7aWIpU\r\nR1ajlOrNvJuhxzwI03/e/jehIpO0h7WQG9A=\r\n=hArj\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220525.147c370.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220525.147c370.tgz","_integrity":"sha512-gRRPayVfSvWw4bBQMGuaI3lys+pXsc7DhsDdDA6Q/XxcKtZ/+NG7kVvNneRQuDcl3/DF8pXCnyAkuBC+e5n3zA==","repository":"microsoft/botframework-cli","_npmVersion":"8.3.1","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220525.147c370","@microsoft/bf-orchestrator":"4.16.0-dev.20220525.147c370"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220525.147c370_1653463382449_0.2746080036161529","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220526.695cee5":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220526.695cee5","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220526.695cee5","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5a00fdf51daf6831646d1f5ed851d31537310a6f","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220526.695cee5.tgz","fileCount":36,"integrity":"sha512-Q8n7IR026JP7NMXraRd98xcrvOzFb2xRJFilvtZQqQQsoHvFEv05pcxkEFLL57ESp3fE0na9gLbL08fk9r1zuQ==","signatures":[{"sig":"MEQCIBwl4KMQJwtDNUD2M1hd3zdGNOVYE8Gm76SE9ntQWHJnAiBBycEUCr2Y4mX/ut7aHJs78Qr+3NIMzSdci2jQ4RT7yQ==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJijypQACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmqxIg//Rm/PJUtsfR5FOkWYKN28+RgFsAuhsd6JePyR8e2lWPdQYTbV\r\nIqISbz2pbh/UPFwufGyLIaPXQuKkkoJ7QPGOs6yDzKLUbWSI8PddLQtOtzn7\r\nxX3NPrKB639BMy6uWySD1l+HipNIXiE0FP4MArme7Xz7C/txWHw87+XTPkM+\r\nb5jmSMt0teCw0/MR7qYdy0gegptbChmHevORoE47+OTARaLva/UjUwlQ986+\r\nLzy5U3Tt1WauIVIXuWPpM/VFuO79jbUqA+qL0VzlWGf65M1PgfOxqTRzT8AU\r\nyUODKMw1BpphBHQP7vND9z/x6wiKb+eVOUT0wifCZ9/VZ17Meq9aptsIyD8y\r\nhZbE/5oY8kUUWehqq8Wnz1czS14CBdQVrylYfIyJAs6dF2boa6yQQJ8V0FxC\r\nWj/tDuq1ra+gX/m2jOrcHjlB5fMyexI1ZCP0PKp8aJ5ODdXl6877lLgrlr3j\r\npuGaokyCGStEeOPqBQJQIaFwnNfbk4v5o6UucXivOVU1y6phVO2VJzM2hc9g\r\nwJHfhfMxUxiF+M9p4FeT7tt/+hkD04uDiOArgDcuMb9ayFsNTrot/QVshZA2\r\nKUracErn6b2BSQqFWYCiZo9yTVOE8YHNnY4JfAV500DMw9rcYd8C8G/BUHqu\r\ngJSafreUoPJcsCsIqBTuVp6eBKdxP8PnGpA=\r\n=wmG2\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220526.695cee5.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220526.695cee5.tgz","_integrity":"sha512-Q8n7IR026JP7NMXraRd98xcrvOzFb2xRJFilvtZQqQQsoHvFEv05pcxkEFLL57ESp3fE0na9gLbL08fk9r1zuQ==","repository":"microsoft/botframework-cli","_npmVersion":"8.3.1","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220526.695cee5","@microsoft/bf-orchestrator":"4.16.0-dev.20220526.695cee5"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220526.695cee5_1653549647771_0.34045930119310164","host":"s3://npm-registry-packages"}},"4.16.0-dev.20220725.a0ab7ca":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20220725.a0ab7ca","keywords":["oclif-plugin"],"author":"Microsoft","license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20220725.a0ab7ca","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":"https://github.com/microsoft/botframework-cli/issues","os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"1855116fcdf42ae23b6e73ca85f96840f94124f8","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20220725.a0ab7ca.tgz","fileCount":36,"integrity":"sha512-L4jDVv3BUaM2cHWnLA4sw+kOcLXLBEKjHM5APNs2HCP3QloB8N9h3fhixiCFv0SZMMGPNqeRkTuKY0+3nZsaAA==","signatures":[{"sig":"MEUCIQCZ5675LzKgtbP7jE8RqtmL1nceZgGrE+9Z987txNbOfAIgeCHA2VJhIuTevsmTumuN5Op/yNaWnrkVXbDn3RMpNlU=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJi3kTAACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmpfwA/+ORjk5jgkSpyvNC2kWRkxUlobloctPJrAfK0XvEXhdJiMukIw\r\nhNzCBoiBBE6nyG4MeVD5aU+5+zM26hLw7uZv7A1MvRft20/GF8dGNTaDRfw4\r\nzi8tC3Jfr4WZKi0G9ZeIGsal2/P+HflRnfrWV2zoukwH2E6DaEuG+/YeH69f\r\nr5+jG5Pbrm8Md66iZH2Q3rVw8MpwWuWV6tf3omK/Bmf4yem05VDh58nybx4T\r\nEfauvJXJmuEJnYGrnVcw/wiHbN5kNrSR+IrZApyyarEYjXHFFeFO4IFkd3tE\r\nx/HgaPrf7AziuPBh6SwHCdQ2kvBy27Zfchep/jz6GlLG3DMG92T9AGhmYVGP\r\nXXIYNtJ/e7SehaOiX1mXftRzD6+jRh9pQ25+tNIkr3R28bHE4ZDYgV3iGU9Q\r\n6nPiVTxMURkqaSR7J0+u0wqSJAAxbxUY2cBfp3FqjCC1QwIwlqEYOvYWCYKo\r\nMApK2ajsdqMtX4XU9v9JXisJOac8FrzWt+oy1UIK3jLpJWaC1mpnwbEB2Qd3\r\nkTlQxzb24H3aXYdqsrDtAdZ4i5JissxIue9ykb4eTQXd+6CImmvvu5BitW8v\r\nAVSKAXjjYwp9v/NphLEZXfur6i1u+cKOQQRsQvnySS3LLem05p7Q0vsB+6pZ\r\nEAqD5aLLOCeIqpvozyLFc+97wiPiWBHFzh4=\r\n=qzw0\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220725.a0ab7ca.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20220725.a0ab7ca.tgz","_integrity":"sha512-L4jDVv3BUaM2cHWnLA4sw+kOcLXLBEKjHM5APNs2HCP3QloB8N9h3fhixiCFv0SZMMGPNqeRkTuKY0+3nZsaAA==","repository":"microsoft/botframework-cli","_npmVersion":"8.3.1","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.14.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20220725.a0ab7ca","@microsoft/bf-orchestrator":"4.16.0-dev.20220725.a0ab7ca"},"_hasShrinkwrap":false,"devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20220725.a0ab7ca_1658733760138_0.9208712628920486","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221107.64f07e4":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20221107.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20221107.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"5f81348bbfa09db2b2d2f1fac9c684c8724ece0c","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20221107.64f07e4.tgz","fileCount":36,"integrity":"sha512-JAhFzJFUE7KtrJtAC7oFxXM89sktbi4GQhqx+m+7rKWjeeuLvnk1ugKOilAPYsAcdwbGdIMAf/NuPeyxUyXRCw==","signatures":[{"sig":"MEQCIFpmD3uokc/ea/Zhm2aFjh2PxoCKvvVh/uf+q5PNSyg1AiBFLxVl76crinXcuITzVdZldiF9giyhjeediH3j/8L06Q==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjaMCSACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmp17g/8CNT47GGBzF35NsjPxNrlLrD14CEY+7M/Q31UnnDv3Qd7Jnhv\r\nIfS6v+xGLGPu5YjFLaOmSOAejpo4A8SHOGKbF0VTppi7kIWLWlNXRrKgnnVD\r\nCvQSryvNy8AGJb0smZZ2Z2uvVxEKZ4T9v5JCOsO13YIAKFCzpvwUs6f/8FON\r\nEsIGQojNEB+WtdezqlZKqCSYWqOoLXjL7IVz6pqRwn1KFgOzV4wMqtspbDdE\r\n5U/Ch41CVobxjIGLZxxKFLxsiU6q+EG1HMXR6MwsynSqyVnWv4Bt4Q0FJfOx\r\nYv8eCyme7dgDEjphj3mdBMYQy3WlRRtYo/T1m0tZCRkAq/BKrULc6jab/9Qp\r\nFQtB/VYHfgIzr/bbmCrpxZoMDKxhOVpA2JXlSU7wXSxwHAHZgchfNAyqcKJr\r\nHCphNkc/ubiE/HpYF6raiVx1y8UrcndFrXtt9eWN8G9aG7Wp0ymI7v8hAgxH\r\neFOTdlLMExJzsZLNJYBIo7L4I3uGeA+Yz2P2Og+1iYT/9BizsjIS/t8CMbvq\r\nQZFrcCc3V3eNhR9vdsd7GGpagtEHpc/lKyQxXqDeum5B6MpbffOg9ukM7kS3\r\no6E2tBQanvW8S57A87TPLMUk2rEYxriCAENxGlOyecFBUCHFQCyB5lgo177q\r\nhhyxzQOD4ikCc6pKzhrcKjq8qU+jlcbM6rE=\r\n=+eiy\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221107.64f07e4.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221107.64f07e4/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221107.64f07e4.tgz","_integrity":"sha512-JAhFzJFUE7KtrJtAC7oFxXM89sktbi4GQhqx+m+7rKWjeeuLvnk1ugKOilAPYsAcdwbGdIMAf/NuPeyxUyXRCw==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"8.19.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20221107.64f07e4","@microsoft/bf-orchestrator":"4.16.0-dev.20221107.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20221107.64f07e4_1667809426344_0.37667558217471697","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221111.64f07e4":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20221111.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20221111.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"d78dc5c39f74288087bf61e2a41a9a0401ed517e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20221111.64f07e4.tgz","fileCount":36,"integrity":"sha512-3pH/iAxoZp0CP2U+S5t4nkJBaSkKYPd9PfLEeM9NgUgMMMpieZKynhMlfc982hDUuKUPcfmL2p9qbn2/5Ntliw==","signatures":[{"sig":"MEUCIQDxYZMMRLi2K3a1ZfEMwKXdOhaRQolMFWUkVevaR6kAKAIgdLQ9Fc8jFXz7IsrDuykAicSG7+y6l35WT1lp3K2oAbg=","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjbtbaACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2VmrxAw/+MjA4qJFhS0sfxXDx7oC4jmKWCN/AZL1K5BGSVyrVYFV0y7sI\r\n6uEI5obFeSrL+P9tC10q2Tngh2uI5Yiq2pl1Q8aP2H4SDIDBBOENMYgTmPz6\r\nm4qDBrGTdl8pm2APf5PbT8/UIfd2AnfAMpA/dUXz5a5tNgriAHtP8g//cQb7\r\n0ZTwW33RDZ8sJtiz4Tbehgp/bEq+dMSqDiWO5/4i9YgvV5eBkDyqLKYeDx7n\r\nGzHrh1VcKripp/j2SPNYN5shJ/j1FiI8QpULQRItTyrloTmJNWDnXzAlcOI9\r\n4erwXNMYkp/wV23rgZgL6aFO8eGfY3z8jrMxQcDz8v/wc8vemHhhM/bv2ycp\r\nN3mC/TQT4dkT4zobPbsSRK4Eg47SmH3g1h8S96/pVecUc8q2j502Ybv80PeN\r\nApTsyKHtYCj8xxR+kXJzO8UY8JEeOaX/IxZ0ruroZoUxVicZ1YI3Qqd4A3UQ\r\nc2t0ALV4hdmeUciakHAtwqM0/GV43ZZsUmHb8Io8vx/pR0TDBWBBlzGGr1yD\r\nWfQFK+90BggDq1lekJW1UOpAu76V7SdRNkj+bblPtE8/reDmMg0VdHFUzfoK\r\nhkXDvaw/VaCMj0KYNfb7oh8mCeIfVV3xuq1a3oqJgLj157mopQRdDWL8nzXI\r\n0TVadcctHYIhyoA9quvID6HqQYXejbin8LY=\r\n=7TXo\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221111.64f07e4.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221111.64f07e4/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221111.64f07e4.tgz","_integrity":"sha512-3pH/iAxoZp0CP2U+S5t4nkJBaSkKYPd9PfLEeM9NgUgMMMpieZKynhMlfc982hDUuKUPcfmL2p9qbn2/5Ntliw==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"8.19.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20221111.64f07e4","@microsoft/bf-orchestrator":"4.16.0-dev.20221111.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20221111.64f07e4_1668208346031_0.07222382511760772","host":"s3://npm-registry-packages"}},"4.16.0-dev.20221114.64f07e4":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20221114.64f07e4","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20221114.64f07e4","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"bdfa9bada9d923acba77f06727308c285e165ba9","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20221114.64f07e4.tgz","fileCount":36,"integrity":"sha512-R5W1Fwu/y8KK6lpOrHxAJhFxFf5G285xyWH3XPgxRUbiL4/2Rkd0IhXViUDBSxcxn/hh+TOUDEPHJpU7NGEF+w==","signatures":[{"sig":"MEQCIGMQeG+2TI/2O+5rM5m7JxNbxmfFNpWEUNlJBgwQQSnTAiA3gOSUCz21vj9TJXPDaDzpAQpuHye5xWfhTXLeFSz+Zw==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018,"npm-signature":"-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v4.10.10\r\nComment: https://openpgpjs.org\r\n\r\nwsFzBAEBCAAGBQJjcfsfACEJED1NWxICdlZqFiEECWMYAoorWMhJKdjhPU1b\r\nEgJ2Vmr0HA//S3R+gcKZQyb4U43QwTS0mSNAFjXJ4CRcMuanMar+Iorap58t\r\nUg/MgavvS2zkPLhePouQYjm8C6hdNmPQUJxedBNipcDm6/uuHcZ6WHhjc+gr\r\n+DclFQp1IXW67WQkAEuidHqfE7FrHS02Zdwsix+EZajeB+MYT/Z7cGULOJQ/\r\nAosnfzRColR5gQoTWLEXZxmuiTL0VPFz+kL+qw5xiRp5jWPwnK6BtiUPnWyw\r\nqSo7mP5yhspqzk5tykhF/fWIbewRZM6A7zwwG++YCvjGlVfQaQjB3wKdmFF9\r\niDa9zfDEzWWqN1ye77fckQjl8p+vBPxcZrNwZ611SRDBmr5NLh6F/BI8c7EF\r\niZG/vHbWMDGWFo/9YgxQUI6poZn9R/q/GqB1Ev0rhOdOQXhNtJ3M5O3D+urW\r\nnuhQ1c0y4Dzj6BhLAW0cZMGpWKOnt9r0Qx1VN1TePrbHyewvnGgg+1oIhGCd\r\nQPfv72+AROXIs5tOCwj0ikMCBOnfnw7Oc7ao6E8s2DE2Jh9lEoynFeBeHqp3\r\nbCNk++PtT3x+UvMmnCRSTD6kJ3X1TdworSYB1iG/3l8CkgbxhBWhX1325L4N\r\n8E6dWj3T8RM8aKVQqOtQJ/d4PzhkZkWm+PQk4nnujeGYOBNum51oVGhYCgTD\r\nRm3TKOCy4/26yawj04VAQdPfIaT0xSwbsbs=\r\n=IHxN\r\n-----END PGP SIGNATURE-----\r\n"},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221114.64f07e4.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20221114.64f07e4/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20221114.64f07e4.tgz","_integrity":"sha512-R5W1Fwu/y8KK6lpOrHxAJhFxFf5G285xyWH3XPgxRUbiL4/2Rkd0IhXViUDBSxcxn/hh+TOUDEPHJpU7NGEF+w==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"8.19.2","description":"Bot Orchestrator","directories":{},"_nodeVersion":"16.18.0","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20221114.64f07e4","@microsoft/bf-orchestrator":"4.16.0-dev.20221114.64f07e4"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20221114.64f07e4_1668414239754_0.9825142782335925","host":"s3://npm-registry-packages"}},"4.16.0-dev.20231122.4561d55":{"name":"@microsoft/bf-orchestrator-cli","version":"4.16.0-dev.20231122.4561d55","keywords":["oclif-plugin"],"author":{"name":"Microsoft"},"license":"MIT","_id":"@microsoft/bf-orchestrator-cli@4.16.0-dev.20231122.4561d55","maintainers":[{"name":"botframework","email":"botframework@microsoft.com"},{"name":"sgellock","email":"sgellock@microsoft.com"},{"name":"cwhitten","email":"christopherwhitten@gmail.com"},{"name":"microsoft1es","email":"npmjs@microsoft.com"},{"name":"peterinnesmsft","email":"peinnes@microsoft.com"},{"name":"joshgummersall","email":"npm@josh.standardlabs.dev"},{"name":"cmullins","email":"cmullins@gmail.com"}],"homepage":"https://github.com/microsoft/botframework-cli","bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"os":["darwin","linux","win32"],"cpu":["x64","ia32"],"dist":{"shasum":"b7f04f1d5ca01293d455a501d8b77ad8d3468f6e","tarball":"https://registry.npmjs.org/@microsoft/bf-orchestrator-cli/-/bf-orchestrator-cli-4.16.0-dev.20231122.4561d55.tgz","fileCount":36,"integrity":"sha512-SwjrcsAmemDLaVJjIUdT6kQQIVIbHsrM2R6mpLsdzZU8FtxNfH05gaj9grNJrluZCpvbh8bNzSg2S7hwtp+GkA==","signatures":[{"sig":"MEQCIBeSgh9g5i4+UOtPrTqKZ6qF9Ru3IOs6sQ5/jWv+AS6UAiAPG0a+QxjMSs8ZkWAbExf9jybU+jEaXzf84o2QrRIy0w==","keyid":"SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA"}],"unpackedSize":131018},"_from":"file:D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20231122.4561d55.tgz","oclif":{"bin":"bf","commands":"./lib/commands","devPlugins":["@oclif/plugin-help"]},"readme":"@microsoft/bf-orchestrator-cli\r\n======================\r\n\r\nThis package is intended for Microsoft use only and should be consumed through @microsoft/botframework-cli. It is not designed to be consumed as an independent package.\r\n\r\n[![oclif](https://img.shields.io/badge/cli-oclif-brightgreen.svg)](https://oclif.io)\r\n[![Version](https://img.shields.io/npm/v/@microsoft/bf-orchestrator-cli.svg)](https://npmjs.org/package/@microsoft/bf-orchestrator-cli)\r\n\r\nOrchestrator CLI is a replacement of the [Dispatch CLI](https://github.com/microsoft/botbuilder-tools/tree/master/packages/Dispatch). Create and evaluate Orchestrator model used to arbitrate across multiple bot modules such as LUIS models, QnA knowledge bases and others.\r\n\r\n# Relevant docs\r\n- [interactive command document][1]\r\n- [test command document][2]\r\n\r\n# Commands\r\n\r\n\n* [`bf orchestrator`](#bf-orchestrator)\n* [`bf orchestrator:add`](#bf-orchestratoradd)\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\n* [`bf orchestrator:query`](#bf-orchestratorquery)\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\n* [`bf orchestrator:test`](#bf-orchestratortest)\n\n## `bf orchestrator`\n\nDisplay Orchestrator CLI available commands\n\n```\nUSAGE\n $ bf orchestrator\n\nOPTIONS\n -h, --help Orchestrator commands help\n```\n\n_See code: [src\\commands\\orchestrator\\index.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\index.ts)_\n\n## `bf orchestrator:add`\n\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:add\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\n -h, --help Orchestrator add command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\n -m, --model=model Path to Orchestrator model directory.\n\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\n working directory.\n\n -t, --type=type Type of input (luis/qna/file).\n\n -v, --version=version Applies only for type=luis, LUIS app version\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --endpoint=endpoint LUIS/QnAMaker endpoint.\n\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\n --routingName=routingName Routing name, default to file name.\n\nEXAMPLE\n\t\n $ bf orchestrator:add \t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \n --endpoint \n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\n```\n\n_See code: [src\\commands\\orchestrator\\add.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\add.ts)_\n\n## `bf orchestrator:basemodel:get`\n\nGets Orchestrator base model\n\n```\nUSAGE\n $ bf orchestrator:basemodel:get\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator basemodel:get command help\n\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\n directory.\n\n -v, --verbose Enable verbose logging\n\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\n subfolder of the output path.\n\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\n not specified, default model will be downloaded.\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\get.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\basemodel\\get.ts)_\n\n## `bf orchestrator:basemodel:list`\n\nLists all Orchestrator base model versions\n\n```\nUSAGE\n $ bf orchestrator:basemodel:list\n\nOPTIONS\n -h, --help Orchestrator basemodel:list command help\n -r, --raw Optional. Raw output\n --all Optional. Display all models\n```\n\n_See code: [src\\commands\\orchestrator\\basemodel\\list.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\basemodel\\list.ts)_\n\n## `bf orchestrator:build`\n\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\n\n```\nUSAGE\n $ bf orchestrator:build\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator build command help\n -i, --in=in Path to lu file or folder with lu files.\n -m, --model=model Path to Orchestrator model.\n\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\n working directory.\n\n --dialog Generate multi language or cross train Orchestrator recognizers.\n\n --luconfig=luconfig Path to luconfig.json.\n```\n\n_See code: [src\\commands\\orchestrator\\build.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\build.ts)_\n\n## `bf orchestrator:create`\n\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\n\n```\nUSAGE\n $ bf orchestrator:create\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help Orchestrator create command help\n\n -i, --in=in The path to source label files from where orchestrator example file will be created\n from. Default to current working directory.\n\n -m, --model=model Path to Orchestrator base model directory.\n\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\n working directory.\n\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\n\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\n```\n\n_See code: [src\\commands\\orchestrator\\create.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\create.ts)_\n\n## `bf orchestrator:interactive`\n\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\n\n```\nUSAGE\n $ bf orchestrator:interactive\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\n -o, --out=out Optional Directory where analysis and output files will be placed.\n\nEXAMPLE\n\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n --model=./path/to/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\interactive.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\interactive.ts)_\n\n## `bf orchestrator:query`\n\nQuery Orchestrator base model and a snapshot/train file\n\n```\nUSAGE\n $ bf orchestrator:query\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\n\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\n all predictions.\n\n -m, --model=model (required) Path to Orchestrator base model directory.\n\n -q, --query=query (required) Query string to predict.\n\nEXAMPLE\n\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\n```\n\n_See code: [src\\commands\\orchestrator\\query.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\query.ts)_\n\n## `bf orchestrator:remove`\n\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\n\n```\nUSAGE\n $ bf orchestrator:remove\n\nOPTIONS\n -d, --debug\n -h, --help Orchestrator remove command help\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\n -t, --type=type Type of input (luis/qna/file).\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\n\nEXAMPLE\n\t\n $ bf orchestrator:remove \t\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \n $ bf orchestrator:remove -t qna --id QNA_KB \n $ bf orchestrator:remove -t file -i FILE_PATH\n```\n\n_See code: [src\\commands\\orchestrator\\remove.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\remove.ts)_\n\n## `bf orchestrator:test`\n\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\n\n```\nUSAGE\n $ bf orchestrator:test\n\nOPTIONS\n -d, --debug\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\n -h, --help show CLI help\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\n\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\n the \"assessment\" mode.\n\n -o, --out=out (required) Directory where analysis and output files will be placed.\n\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\n (e.g., crosss-valiaton) files.\n\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\n\nDESCRIPTION\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\n a previously generated Orchestrator .blu snapshot/train file,\n and create a detailed train/test evaluation report.\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\n on a previously generated Orchestrator .blu snapshot/train file.\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\n create an evaluation report. This mode can evaluate predictions produced by\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\n the specification detailed below.\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\n a data instance can be built directly from the features and put in place of text\n in a prediction and ground-truth file.\n\n The 'test' mode is activated if there is a '--test' argument set for a test file.\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\n\nEXAMPLE\n\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \n --model=./path/to/model/directory\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \n --out=./path/to/output/folder/\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \n [--model=./path/to/model/directory]\n```\n\n_See code: [src\\commands\\orchestrator\\test.ts](https://github.com/microsoft/botframework-cli/blob/v4.16.0-dev.20231122.4561d55/src\\commands\\orchestrator\\test.ts)_\n\r\n* [`bf orchestrator`](#bf-orchestrator)\r\n* [`bf orchestrator:add`](#bf-orchestratoradd)\r\n* [`bf orchestrator:basemodel:get`](#bf-orchestratorbasemodelget)\r\n* [`bf orchestrator:basemodel:list`](#bf-orchestratorbasemodellist)\r\n* [`bf orchestrator:build`](#bf-orchestratorbuild)\r\n* [`bf orchestrator:create`](#bf-orchestratorcreate)\r\n* [`bf orchestrator:interactive`](#bf-orchestratorinteractive)\r\n* [`bf orchestrator:query`](#bf-orchestratorquery)\r\n* [`bf orchestrator:remove`](#bf-orchestratorremove)\r\n* [`bf orchestrator:test`](#bf-orchestratortest)\r\n\r\n## `bf orchestrator`\r\n\r\nDisplay Orchestrator CLI available commands\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator\r\n\r\nOPTIONS\r\n -h, --help Orchestrator commands help\r\n```\r\n\r\n_See code: [src/commands/orchestrator/index.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/index.ts)_\r\n\r\n## `bf orchestrator:add`\r\n\r\nAdd examples from .lu/.qna/.json/.blu files, LUIS app(s) and QnaMaker kb(s) to Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:add\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -f, --force If --out flag is provided with the path to an existing file, overwrites that file.\r\n -h, --help Orchestrator add command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -k, --key=key LUIS authoring key or QnAMaker service key if type = luis/qna.\r\n -m, --model=model Path to Orchestrator model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator example file will be placed. Default to current\r\n working directory.\r\n\r\n -t, --type=type Type of input (luis/qna/file).\r\n\r\n -v, --version=version Applies only for type=luis, LUIS app version\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --endpoint=endpoint LUIS/QnAMaker endpoint.\r\n\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\n --routingName=routingName Routing name, default to file name.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:add \t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --snapshot ./path/to/snapshot/ --out ./path/to/output/\t\r\n $ bf orchestrator:add --in ./path/to/file/ --out ./path/to/output/ --model ./path/to/model/directory\r\n $ bf orchestrator:add -t luis --id LUIS_APP_ID --version LUIS_APP_VERSION --key LUIS_KEY --routingName l_Weather \r\n --endpoint \r\n $ bf orchestrator:add -t qna --id QNA_KB --key QNA_KB_SERVICE_KEY --routingName q_kb\r\n```\r\n\r\n_See code: [src/commands/orchestrator/add.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/add.ts)_\r\n\r\n## `bf orchestrator:basemodel:get`\r\n\r\nGets Orchestrator base model\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:get\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator basemodel:get command help\r\n\r\n -o, --out=out Optional. Path to where Orchestrator base model will be saved to. Default to current working\r\n directory.\r\n\r\n -v, --verbose Enable verbose logging\r\n\r\n --getEntity Optional. Download default entity model at the same time, which will be placed in the entity\r\n subfolder of the output path.\r\n\r\n --versionId=versionId Optional. Base model version to download -- reference basemodel:list output for options. If\r\n not specified, default model will be downloaded.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/get.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/get.ts)_\r\n\r\n## `bf orchestrator:basemodel:list`\r\n\r\nLists all Orchestrator base model versions\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:basemodel:list\r\n\r\nOPTIONS\r\n -h, --help Orchestrator basemodel:list command help\r\n -r, --raw Optional. Raw output\r\n --all Optional. Display all models\r\n```\r\n\r\n_See code: [src/commands/orchestrator/basemodel/list.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/basemodel/list.ts)_\r\n\r\n## `bf orchestrator:build`\r\n\r\nCreates Orchestrator snapshot file and Orchestrator dialog definition file (optional) for each lu file in input folder.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:build\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator build command help\r\n -i, --in=in Path to lu file or folder with lu files.\r\n -m, --model=model Path to Orchestrator model.\r\n\r\n -o, --out=out Path where Orchestrator snapshot/dialog file(s) will be placed. Default to current\r\n working directory.\r\n\r\n --dialog Generate multi language or cross train Orchestrator recognizers.\r\n\r\n --luconfig=luconfig Path to luconfig.json.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/build.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/build.ts)_\r\n\r\n## `bf orchestrator:create`\r\n\r\nCreate orchestrator snapshot (.blu) file from .lu/.qna/.json/.tsv/.dispatch files, which represent bot modules\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:create\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help Orchestrator create command help\r\n\r\n -i, --in=in The path to source label files from where orchestrator example file will be created\r\n from. Default to current working directory.\r\n\r\n -m, --model=model Path to Orchestrator base model directory.\r\n\r\n -o, --out=out Path where generated Orchestrator snapshot file will be placed. Default to current\r\n working directory.\r\n\r\n --hierarchical Add hierarchical labels based on .lu/.qna file name. Resulting snapshot file will\r\n contain.lu/.qna file name as labels instead of the intents defined in the .lu file(s).\r\n\r\n --refresh Refetch LUIS app(s)/QnAMaker kb(s) previously added and recreate Orchestrator snapshot.\r\n```\r\n\r\n_See code: [src/commands/orchestrator/create.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/create.ts)_\r\n\r\n## `bf orchestrator:interactive`\r\n\r\nReal-time interaction with Orchestrator model and analysis. Can return score of given utterance using previously created orchestrator examples\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:interactive\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -l, --in=in Optional path to a previously created Orchestrator .blu file.\r\n -m, --model=model (required) Directory or hosting Orchestrator config and base model files.\r\n -o, --out=out Optional Directory where analysis and output files will be placed.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:interactive --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n --model=./path/to/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/interactive.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/interactive.ts)_\r\n\r\n## `bf orchestrator:query`\r\n\r\nQuery Orchestrator base model and a snapshot/train file\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:query\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator snapshot (.blu file).\r\n\r\n -l, --limit=limit (optional) Limit of number of predictions. Default to 3. Less or equal to 0 for listing\r\n all predictions.\r\n\r\n -m, --model=model (required) Path to Orchestrator base model directory.\r\n\r\n -q, --query=query (required) Query string to predict.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:query --in=./path/to/snapshot/file --query=hi --model=./path/to/base/model/directory\r\n```\r\n\r\n_See code: [src/commands/orchestrator/query.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/query.ts)_\r\n\r\n## `bf orchestrator:remove`\r\n\r\nRemove examples from LUIS app(s), QnaMaker kb(s) or .lu/.qna/.json files from Orchestrator snapshot file.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:remove\r\n\r\nOPTIONS\r\n -d, --debug\r\n -h, --help Orchestrator remove command help\r\n -i, --in=in Path to example file (.lu/.qna/.json/.blu).\r\n -t, --type=type Type of input (luis/qna/file).\r\n --id=id LUIS app id or QnAMaker kb id if type = luis/qna.\r\n\r\nEXAMPLE\r\n\t\r\n $ bf orchestrator:remove \t\r\n $ bf orchestrator:remove -t luis --id LUIS_APP_ID \r\n $ bf orchestrator:remove -t qna --id QNA_KB \r\n $ bf orchestrator:remove -t file -i FILE_PATH\r\n```\r\n\r\n_See code: [src/commands/orchestrator/remove.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/remove.ts)_\r\n\r\n## `bf orchestrator:test`\r\n\r\nThe \"test\" command can operate in three modes: test, evaluation, assessment.\r\n\r\n```\r\nUSAGE\r\n $ bf orchestrator:test\r\n\r\nOPTIONS\r\n -d, --debug\r\n -e, --entityModel=entityModel Path to Orchestrator entity base model directory.\r\n -h, --help show CLI help\r\n -i, --in=in (required) Path to a previously created Orchestrator .blu file.\r\n\r\n -m, --model=model Optional directory for hosting Orchestrator config and base model files, not needed for\r\n the \"assessment\" mode.\r\n\r\n -o, --out=out (required) Directory where analysis and output files will be placed.\r\n\r\n -p, --prediction=prediction Optional path to a prediction label file, or comma-separated paths to a collection of\r\n (e.g., crosss-valiaton) files.\r\n\r\n -t, --test=test Optional path to a test file. This option enable the \"test\" mode.\r\n\r\nDESCRIPTION\r\n 1) Test mode: test a collection of utterance/label samples loaded from a test file against\r\n a previously generated Orchestrator .blu snapshot/train file,\r\n and create a detailed train/test evaluation report.\r\n 2) Evaluation mode: create an leave-one-out cross validation (LOOCV) evaluation report\r\n on a previously generated Orchestrator .blu snapshot/train file.\r\n 3) Assessment mode: assess a collection of utterance/label predictions against their ground-truth labels and\r\n create an evaluation report. This mode can evaluate predictions produced by\r\n other NLP or machine learning systems. There is no need for an Orchestrator base model.\r\n Notice that, this mode is generic and can apply to evaluate any ML systems, learners, models,\r\n and scenarios if a user can carefully construct the prediction and grounf-truth files by\r\n the specification detailed below.\r\n Essentially the key to a NLP data instance is a text (utterance, sentence, query, document, etc.), which\r\n is the basis of all the features feeding to a ML model. For other ML systems, the key to\r\n a data instance can be built directly from the features and put in place of text\r\n in a prediction and ground-truth file.\r\n\r\n The 'test' mode is activated if there is a '--test' argument set for a test file.\r\n The 'assessment' mode is activated if there is a '--prediction' argument set for a prediction file.\r\n If there is no '--test' or '--prediction' arguments, then \"test\" command runs on the 'evaluation' mode.\r\n\r\nEXAMPLE\r\n\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --test=./path/to/test/file/ --out=./path/to/output/ \r\n --model=./path/to/model/directory\r\n $ bf orchestrator:test --in=./path/to/ground-truth/file --prediction=./path/to/prediction/file \r\n --out=./path/to/output/folder/\r\n $ bf orchestrator:test --in=./path/to/snapshot/file --out=./path/to/output/folder/ \r\n [--model=./path/to/model/directory]\r\n```\r\n\r\n_See code: [src/commands/orchestrator/test.ts](https://github.com/microsoft/botframework-cli/src/commands/orchestrator/test.ts)_\r\n\r\n\r\n# Orchestrator CLI settings file\r\nMost of bf orchestrator commands create or update a settings file (orchestrator.json) with base model and snapshot file path so that those values could be optional for subsequent commands. For example, after ```bf orchestrator:basemodel:get --out ```, subsequent command (ie bf orchestrator:create) will no longer need to supply the \"--model\" parameter.\r\n\r\n[1]:./docs/interactive-command.md\r\n[2]:./docs/test-command.md\n","engines":{"node":"^10.0.0 || ^12.0.0 || ^13.0.0 || ^14.0.0"},"scripts":{"doc":"npm run build && npm run doc:readme && rimraf oclif.manifest.json","lint":"eslint . --ext .ts","test":"mocha","build":"tsc -b","clean":"rimraf ./.nyc_output ./package-lock.json ./tsconfig.tsbuildinfo","prepack":"tsc -b && oclif-dev manifest","version":"npm run doc:readme && git add README.md","coverage":"nyc npm run test","postpack":"rimraf oclif.manifest.json","posttest":"eslint . --ext .ts --config .eslintrc","doc:readme":"oclif-dev manifest && oclif-dev readme","report:coverage":"npm run coverage && nyc report --reporter=text-lcov | coveralls"},"_npmUser":{"name":"botframework","email":"botframework@microsoft.com"},"_resolved":"D:\\a\\r1\\a\\_botframework-cli-daily\\drop\\microsoft-bf-orchestrator-cli-4.16.0-dev.20231122.4561d55.tgz","_integrity":"sha512-SwjrcsAmemDLaVJjIUdT6kQQIVIbHsrM2R6mpLsdzZU8FtxNfH05gaj9grNJrluZCpvbh8bNzSg2S7hwtp+GkA==","repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"_npmVersion":"9.8.1","description":"Bot Orchestrator","directories":{},"_nodeVersion":"18.18.2","dependencies":{"tslib":"^2.0.3","fs-extra":"~9.0.0","@oclif/config":"^1.14.0","@oclif/errors":"~1.2.2","@oclif/command":"^1.5.19","read-text-file":"~1.1.0","@types/fs-extra":"~8.1.0","@oclif/plugin-help":"^2","@microsoft/bf-cli-command":"4.16.0-dev.20231122.4561d55","@microsoft/bf-orchestrator":"4.16.0-dev.20231122.4561d55"},"_hasShrinkwrap":false,"readmeFilename":"README.md","devDependencies":{"nyc":"^14.1.1","chai":"^4.2.0","mocha":"^6.2.2","sinon":"^9.0.2","eslint":"^5.16.0","globby":"^10.0.2","rimraf":"^3.0.2","ts-node":"^8.6.2","typescript":"^4.0.3","@oclif/test":"^1.2.5","@types/chai":"^4.2.9","@types/node":"^10.17.15","@types/mocha":"^5.2.7","@types/sinon":"^9.0.4","@oclif/dev-cli":"^1.22.2","@oclif/plugin-help":"^2.2.3","eslint-config-oclif":"^3.1.0","eslint-config-oclif-typescript":"^0.1.0"},"_npmOperationalInternal":{"tmp":"tmp/bf-orchestrator-cli_4.16.0-dev.20231122.4561d55_1700671055617_0.5569655289031608","host":"s3://npm-registry-packages"}}},"time":{"created":"2020-08-07T23:22:46.674Z","modified":"2026-07-16T15:43:38.395Z","0.0.0":"2020-08-07T23:22:46.805Z","4.11.0-beta.875403c":"2020-08-12T22:13:25.871Z","4.11.0-beta.6e2953b":"2020-08-12T23:59:27.894Z","4.11.0-beta.6afce3c":"2020-08-13T18:50:20.572Z","4.11.0-beta.20200818.8213b2a":"2020-08-18T20:20:41.696Z","4.11.0-beta.20200821.21f8e76":"2020-08-21T18:47:42.273Z","4.11.0-beta.20200824.6c63bf3":"2020-08-24T07:08:52.462Z","4.11.0-beta.20200904.e677869":"2020-09-04T07:09:15.640Z","4.11.0-beta.20200918.ad209cb":"2020-09-18T07:09:34.506Z","4.11.0-beta.20200919.b7fef29":"2020-09-19T07:08:25.340Z","4.11.0-beta.20200921.2320a67":"2020-09-21T07:08:46.392Z","4.11.0-beta.20200922.a071f5c":"2020-09-22T18:10:46.978Z","4.11.0-beta.20200923.7e94eb2":"2020-09-23T07:09:29.124Z","4.11.0-beta.20201003.a485403":"2020-10-03T07:07:39.052Z","4.11.0-beta.20201006.29a8107":"2020-10-06T07:08:50.820Z","4.11.0-beta.20201008.9bc8859":"2020-10-08T07:10:00.899Z","4.11.0-beta.20201009.5113321":"2020-10-09T07:09:13.765Z","4.11.0-beta.20201010.6819330":"2020-10-10T07:10:22.156Z","4.11.0-beta.20201011.d140767":"2020-10-11T07:09:14.320Z","4.11.0-beta.20201013.20d7917":"2020-10-13T07:22:02.915Z","4.11.0-beta.20201014.ce6aa6c":"2020-10-14T07:11:12.792Z","4.11.0-beta.20201015.008a3a4":"2020-10-15T07:17:33.503Z","4.11.0-beta.20201016.393c6b2":"2020-10-16T07:18:20.784Z","4.11.0-beta.20201022.4350e73":"2020-10-22T07:08:37.021Z","4.11.0-beta.20201023.67f9a0d":"2020-10-23T07:10:14.194Z","4.11.0-beta.20201024.23949ae":"2020-10-24T07:09:05.146Z","4.11.0-beta.20201025.035cdcb":"2020-10-25T07:09:28.540Z","4.11.0-beta.20201026.d8a7aef":"2020-10-26T07:09:37.314Z","4.11.0-beta.20201027.744fa02":"2020-10-27T07:09:43.670Z","4.11.0-beta.20201028.1b0cde3":"2020-10-28T07:09:43.031Z","4.11.0-beta.20201029.e864bc8":"2020-10-29T07:09:01.352Z","4.11.0-beta.20201030.881c37e":"2020-10-30T07:09:20.410Z","4.11.0-beta.20201031.17d6e9e":"2020-10-31T07:08:30.219Z","4.11.0-beta.20201101.07f38db":"2020-11-01T07:09:05.496Z","4.11.0-beta.20201103.f78e0cf":"2020-11-03T07:08:40.266Z","4.11.0-beta.20201104.673f22d":"2020-11-04T07:09:53.284Z","4.11.0-beta.20201105.0cc03df":"2020-11-05T00:14:27.134Z","4.11.0-beta.20201106.87dcae0":"2020-11-06T07:09:21.954Z","4.11.0-beta.20201107.23b4af1":"2020-11-07T07:08:32.916Z","4.11.0-beta.20201111.10062c1":"2020-11-11T07:09:40.562Z","4.11.0-beta.20201113.e85b53d":"2020-11-13T07:09:43.057Z","4.11.0-beta.20201114.d501bf0":"2020-11-14T07:09:18.975Z","4.11.0-beta.20201203.9b18c73":"2020-12-03T07:09:15.792Z","4.11.0-beta.20201203.69f8eac":"2020-12-03T18:23:25.123Z","4.11.0-beta.20201203.cd29e25":"2020-12-03T23:25:46.650Z","4.11.0-beta.20201204.cd29e25":"2020-12-04T07:09:37.926Z","4.11.0-beta.20201208.165a6d6":"2020-12-08T19:25:13.440Z","4.11.0-beta.20201209.165a6d6":"2020-12-09T07:08:51.536Z","4.11.0-beta.20201209.54c74f9":"2020-12-09T21:03:29.761Z","4.11.0-beta.20201210.9534f1b":"2020-12-10T07:09:50.011Z","4.11.0-beta.20201211.b852646":"2020-12-11T18:09:22.845Z","4.11.0-beta.20201212.7d972c3":"2020-12-12T07:09:22.423Z","4.11.0-beta.20210124.468c75a":"2021-01-24T07:08:43.935Z","4.11.0-beta.20210127.476c841":"2021-01-27T07:09:43.404Z","4.11.0-beta.20210128.3d35ae9":"2021-01-28T07:09:19.921Z","4.11.0-beta.20210129.453a0f1":"2021-01-29T07:09:08.504Z","4.11.0-beta.20210130.46cecc7":"2021-01-30T07:08:36.585Z","4.11.0-beta.20210202.f5bfbad":"2021-02-02T07:09:08.943Z","4.11.0-beta.20210203.b0a8b02":"2021-02-03T07:07:46.367Z","4.11.0-beta.20210205.12121c8":"2021-02-05T07:08:05.089Z","4.11.0-beta.20210206.dde4bda":"2021-02-06T07:08:52.179Z","4.11.0-beta.20210207.af5cc18":"2021-02-07T07:08:46.084Z","4.11.0-beta.20210209.a44ee5e":"2021-02-09T07:09:51.763Z","4.11.0-beta.20210210.116dcad":"2021-02-10T07:10:03.377Z","4.11.0-beta.20210211.c895ecf":"2021-02-11T07:09:38.848Z","4.11.0-beta.20210212.2884eed":"2021-02-12T07:09:30.509Z","4.11.0-beta.20210213.4729245":"2021-02-13T07:09:54.071Z","4.11.0-beta.20210214.41d8748":"2021-02-14T07:09:56.844Z","4.12.0-beta.20210217.668cdac":"2021-02-17T07:09:09.962Z","4.12.0-beta.20210218.668cdac":"2021-02-18T00:44:47.385Z","4.12.0-beta.20210223.d42fe4c":"2021-02-23T19:29:52.540Z","4.12.0-beta.20210224.8610f7b":"2021-02-24T07:47:24.525Z","4.12.0-beta.20210224.dd555e8":"2021-02-24T23:28:09.247Z","4.12.0-beta.20210225.6744890":"2021-02-25T07:09:47.825Z","4.12.0-beta.20210227.ba15791":"2021-02-27T07:09:10.648Z","4.12.0-beta.20210310.a728f41":"2021-03-10T17:41:25.739Z","4.12.0-beta.20210311.6018ddf":"2021-03-11T17:34:21.912Z","4.12.0-beta.20210311.98dc244":"2021-03-11T19:39:40.999Z","4.12.0-beta.20210316.cdd0819":"2021-03-16T21:08:04.256Z","4.13.0-beta.20210316.e8ec340":"2021-03-16T22:18:28.096Z","4.12.0-beta.20210322.314475a":"2021-03-22T16:36:40.736Z","4.13.0-rc0":"2021-04-02T00:50:53.082Z","4.14.0-dev.20210402.7e5fa93":"2021-04-02T07:21:46.594Z","4.14.0-dev.20210406.74baacc":"2021-04-06T07:20:21.154Z","4.14.0-dev.20210407.6745788":"2021-04-07T23:47:29.018Z","4.14.0-dev.20210408.393d33d":"2021-04-08T07:20:26.424Z","4.14.0-dev.20210408.1a5070e":"2021-04-08T17:01:10.158Z","4.14.0-dev.20210409.1a5070e":"2021-04-09T07:20:53.873Z","4.13.0-rc1":"2021-04-09T17:30:49.479Z","4.14.0-dev.20210415.161c029":"2021-04-15T07:20:35.786Z","4.14.0-dev.20210415.7e35288":"2021-04-15T19:15:19.110Z","4.14.0-dev.20210415.ab0fc09":"2021-04-15T20:23:40.309Z","4.13.0-rc2":"2021-04-15T22:50:46.513Z","4.14.0-dev.20210416.ab0fc09":"2021-04-16T07:20:36.796Z","4.14.0-dev.20210419.7257bc2":"2021-04-19T18:34:57.731Z","4.13.0":"2021-04-19T22:19:16.764Z","4.14.0-dev.20210420.fc92e9b":"2021-04-20T07:20:41.064Z","4.14.0-dev.20210422.eccdf02":"2021-04-22T07:22:04.182Z","4.13.1":"2021-04-26T21:41:06.948Z","4.14.0-dev.20210427.c0be653":"2021-04-27T07:21:53.956Z","4.13.2":"2021-04-28T22:42:29.414Z","4.14.0-dev.20210429.968ef75":"2021-04-29T15:55:24.420Z","4.13.3":"2021-04-29T20:53:33.319Z","4.14.0-dev.20210430.968ef75":"2021-04-30T07:19:42.318Z","4.14.0-dev.20210511.07bbd05":"2021-05-11T07:22:52.513Z","4.14.0-dev.20210517.4cdd4d9":"2021-05-17T07:23:51.515Z","4.14.0-dev.20210519.92dfb46":"2021-05-19T07:21:23.978Z","4.14.0-dev.20210521.64e1d40":"2021-05-21T07:22:53.621Z","4.14.0-dev.20210602.be805a8":"2021-06-02T07:25:58.551Z","4.14.0-dev.20210604.3f9ee15":"2021-06-04T07:23:47.874Z","4.14.0-dev.20210623.37f59ec":"2021-06-23T07:31:23.839Z","4.14.0-dev.20210624.eac7a60":"2021-06-24T07:22:40.267Z","4.14.0-dev.20210625.132e0cd":"2021-06-25T07:21:47.314Z","4.14.0-dev.20210629.e97e6b0":"2021-06-29T07:22:23.708Z","4.14.0-rc0":"2021-06-29T18:06:29.835Z","4.14.0-dev.20210630.8078cc6":"2021-06-30T07:21:50.725Z","4.15.0-dev.20210701.be9cd6c":"2021-07-01T07:19:59.632Z","4.15.0-dev.20210702.cbf708d":"2021-07-02T07:21:49.605Z","4.15.0-dev.20210707.cf471cf":"2021-07-07T07:20:35.810Z","4.14.0":"2021-07-07T19:27:23.053Z","4.15.0-dev.20210708.f16ca8f":"2021-07-08T07:19:47.541Z","4.14.1":"2021-07-08T18:41:00.489Z","4.15.0-dev.20210712.eb38bc0":"2021-07-12T07:19:42.101Z","4.15.0-dev.20210715.ead1751":"2021-07-15T07:19:45.679Z","4.15.0-dev.20210723.bf1f054":"2021-07-23T07:24:07.673Z","4.15.0-dev.20210803.40549ca":"2021-08-03T07:22:13.566Z","4.15.0-dev.20210805.6321dcc":"2021-08-05T07:20:41.733Z","4.15.0-dev.20210914.7a936c1":"2021-09-14T07:21:23.428Z","4.15.0-dev.20210917.b688652":"2021-09-17T07:24:05.443Z","4.15.0-dev.20210920.e4d38fc":"2021-09-20T07:24:19.452Z","4.15.0-dev.20210921.fcb2cdb":"2021-09-21T07:23:11.683Z","4.15.0-dev.20210922.c1c0528":"2021-09-22T07:22:24.704Z","4.15.0-dev.20210924.20a3f98":"2021-09-24T07:19:30.524Z","4.15.0-dev.20210928.a9a92a5":"2021-09-28T07:22:57.186Z","4.15.0-dev.20211004.edcf63e":"2021-10-04T07:25:23.589Z","4.15.0-rc0":"2021-10-15T22:52:31.079Z","4.16.0-dev.20211020.406dfef":"2021-10-20T07:23:58.397Z","4.16.0-dev.20211101.bed3c05":"2021-11-01T17:35:45.634Z","4.16.0-dev.20211102.bed3c05":"2021-11-02T07:24:53.844Z","4.16.0-dev.20211102.3f7a67a":"2021-11-02T17:48:28.119Z","4.15.0-rc1":"2021-11-02T23:17:37.929Z","4.16.0-dev.20211103.3f7a67a":"2021-11-03T07:20:29.331Z","4.16.0-dev.20211105.90655c4":"2021-11-05T07:25:04.744Z","4.15.0-rc2":"2021-11-05T18:21:38.275Z","4.15.0-rc3":"2021-11-10T23:40:35.392Z","4.16.0-dev.20211111.15b5c37":"2021-11-11T08:22:37.309Z","4.15.0":"2021-11-16T17:11:52.535Z","4.16.0-dev.20211117.19ff452":"2021-11-17T08:23:55.468Z","4.16.0-dev.20211220.6972973":"2021-12-20T08:24:19.296Z","4.16.0-dev.20220104.33b5e0a":"2022-01-04T08:24:02.338Z","4.16.0-dev.20220105.380511c":"2022-01-05T21:10:35.425Z","4.16.0-dev.20220105.c44b989":"2022-01-06T00:02:16.975Z","4.16.0-dev.20220106.c44b989":"2022-01-06T08:21:30.415Z","4.16.0-dev.20220110.b4d497d":"2022-01-10T08:22:44.545Z","4.16.0-dev.20220120.e933dfc":"2022-01-20T08:25:00.931Z","4.16.0-dev.20220228.5bb413a":"2022-02-28T08:21:13.764Z","4.16.0-dev.20220525.147c370":"2022-05-25T07:23:02.596Z","4.16.0-dev.20220526.695cee5":"2022-05-26T07:20:47.992Z","4.16.0-dev.20220725.a0ab7ca":"2022-07-25T07:22:40.322Z","4.16.0-dev.20221107.64f07e4":"2022-11-07T08:23:46.570Z","4.16.0-dev.20221111.64f07e4":"2022-11-11T23:12:26.153Z","4.16.0-dev.20221114.64f07e4":"2022-11-14T08:23:59.953Z","4.16.0-dev.20231122.4561d55":"2023-11-22T16:37:35.821Z"},"bugs":{"url":"https://github.com/microsoft/botframework-cli/issues"},"author":{"name":"Microsoft"},"license":"MIT","homepage":"https://github.com/microsoft/botframework-cli","keywords":["oclif-plugin"],"repository":{"url":"git+https://github.com/microsoft/botframework-cli.git","type":"git"},"description":"Bot Orchestrator","maintainers":[{"email":"botframework@microsoft.com","name":"botframework"},{"email":"sgellock@microsoft.com","name":"sgellock"},{"email":"christopherwhitten@gmail.com","name":"cwhitten"},{"email":"npmjs@microsoft.com","name":"microsoft1es"},{"email":"microsoft-oss-publishing@microsoft.com","name":"microsoft-oss-releases"},{"email":"peinnes@microsoft.com","name":"peterinnesmsft"},{"email":"cmullins@gmail.com","name":"cmullins"}],"readme":"","readmeFilename":""}